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ABSTRACT 


The  Navy  College  Program  for  Afloat  College  Education  (NCPACE)  is  one  of  the  main 
components  of  the  United  States  Navy’s  Voluntary  Education  (VOLED)  program, 
offering  college  courses  and  remedial  academic  skill  modules  to  sailors  on  sea  duty  or 
stationed  in  remote  locations. 

This  thesis  predicts  the  likelihood  of  NCPACE  course  completion  by  course  and 
individual  participant  characteristics  using  a  logistic  regression  model.  We  found  that 
participants  who  take  distance  learning-based  and  mathematics  courses  have  lower 
predicted  odds  of  succeeding,  while  participants  with  higher  Armed  Forces  Qualification 
Test  (AFQT)  scores  and  levels  of  education  have  higher  predicted  odds  of  succeeding. 
Some  variation  was  noticed  between  unit  vessel  types  and  school. 

This  thesis  also  evaluates  the  likelihood  of  attempting  and  successfully 
completing  a  subsequent  course  conditional  on  the  outcome  of  the  first  course. 
Successful  completion  of  the  first  course  is  positively  associated  with  an  enrollment  in  a 
subsequent  course  in  addition  to  the  successfully  completion  of  that  course. 

Lastly,  this  thesis  examines  promotion,  extension  and  reenlistment  outcomes  for 
first-term  NCPACE  participants  with  48-month  contracts.  We  found  that  those  who 
successfully  completed  at  least  one  course  are  predicted  to  be  more  likely  to  promote  to 
E5  and  are  predicted  to  be  slightly  less  likely  to  reenlist  in  the  Navy. 
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EXECUTIVE  SUMMARY 


Department  of  Defense  (DoD)  Directive  1322.08E  states  that  it  is  DoD  policy  to  establish 
and  maintain  voluntary  education  (VOLED)  programs  that  are  available  to  all  active  duty 
personnel,  regardless  of  duty  location.  VOLED  programs  are  defined  as  continuing, 
adult,  or  postsecondary  education  programs  that  service  members  elect  to  participate  in 
during  their  off-duty  time  and  are  directed  to  provide  educational  opportunities 
comparable  to  those  that  are  available  to  civilians. 

The  Navy  College  Program  for  Afloat  College  Education  (NCPACE)  is  one  of  the 
main  components  of  the  United  States  Navy’s  VOLED  program.  NCPACE  offers 
college  courses  and  remedial  academic  skill  modules  to  sailors  on  sea  duty.  These 
courses  are  delivered  through  electronic  methods  (Technology  PACE)  or  traditional 
classroom  instruction  (Instructor  PACE).  Courses  are  tuition-free,  and  sailors  pay  only 
costs  associated  with  college  coursework  educational  materials.  All  undergraduate 
courses  are  from  institutions  affiliated  with  the  Service  members  Opportunity  College 
degree  program  for  the  Navy,  ensuring  sailors  the  opportunity  to  transfer  credits  and 
complete  degrees. 

A  recent  study  indicates  a  significant  rise  in  the  number  and  percentage  of  sailors 
enrolling  in  distance  education  courses  through  the  Navy’s  Tuition  Assistance  (TA) 
program  between  1995  and  2008.  This  growth  in  distance  education  enrollments  has 
been  attributed  to  the  increase  in  operational  tempo  since  September  2001.  Distance 
learning  course  completion  rates  are  reported  as  lagging  behind  traditional  course 
completion  rates  for  all  years  considered,  with  Physical  Science  courses  experiencing  the 
biggest  differences  in  completion  rates  between  instruction  methods  (Woosely,  2009). 

Though  the  study  referenced  above  makes  an  inferential  connection  between 
distance  learning  and  operational  tempo  events,  it  and  other  evaluations  of  military- 
sponsored  continuing  education  programs  done  in  the  past  do  not  directly  address  how 
students  perform  academically  in  operational  environments.  This  thesis  attempts  to  fill 
this  gap  by  evaluating  the  effectiveness  of  the  NCPACE  component  of  VOLED. 


xv 


NCPACE  is  a  good  candidate  for  addressing  the  effectiveness  of  distance  education  and 
academic  success  in  operational  environments  in  general,  as  the  program’s  students  are 
all  on  sea  duty  and  the  program  has  a  distance  education  component.  Identifying  the 
factors  associated  with  successful  NCPACE  course  completions  provides  information 
necessary  to  make  VOLED  programs  more  effective  as  a  whole. 

This  thesis  predicts  the  likelihood  of  NCPACE  course  completions  by  course  and 
individual  participant  characteristics  using  a  logistic  regression  model.  We  found  that 
participants  who  take  distance  learning-based  and  mathematics  courses  have  lower 
predicted  odds  of  succeeding,  while  higher  Armed  Forces  Qualification  Test  (AFQT) 
scores  and  levels  of  education  resulted  in  higher  predicted  odds  of  succeeding.  Some 
variation  was  noticed  between  unit  vessel  types  and  school. 

This  thesis  also  evaluates  the  likelihood  of  attempting  and  successfully 
completing  a  subsequent  course  with  a  consideration  given  to  the  outcome  of  the  first 
course.  Successful  completion  of  the  first  course  is  positively  associated  with 
enrollments  in  subsequent  courses  and  in  successfully  completing  subsequent  ones. 

Lastly,  this  thesis  examines  promotion,  extension  and  reenlistment  outcomes  for 
first-term  NCPACE  participants  with  48-month  contracts.  We  found  that  those  who 
successfully  completed  at  least  one  course  are  predicted  to  be  more  likely  to  promote  to 
E5  and  are  predicted  to  be  slightly  less  likely  to  reenlist  in  the  Navy. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Department  of  Defense  (DoD)  Directive  1322. 08E  states  that  it  is  DoD  policy  to 
establish  and  maintain  voluntary  education  (VOLED)  programs  that  are  available  to  all 
active  duty  personnel,  regardless  of  duty  location.  VOLED  programs  are  defined  as 
continuing,  adult,  or  postsecondary  education  programs  that  service  members  elect  to 
participate  in  during  their  off-duty  time  and  which  are  directed  to  provide  educational 
opportunities  comparable  to  those  that  are  available  to  civilians. 

The  Navy  College  Program  for  Afloat  College  Education  (NCPACE)  is  one  of  the 
main  components  of  the  United  States  Navy’s  VOLED  program.  NCPACE  offers 
college  courses  and  remedial  academic  skill  modules  to  sailors  on  sea  duty  or  in 
designated  geographically  isolated  installations.  What  makes  this  program  important  is 
that  it  is  specifically  geared  towards  Service  members  who  would  otherwise  have 
difficulty  gaining  access  to  continuing  education  resources  due  to  their  physical  location 
and  less-than-predictable  work  schedules.  These  courses  are  delivered  through  electronic 
methods  (Technology  PACE)  or  traditional  classroom  instruction  (Instructor  PACE). 
Courses  are  tuition-free,  and  sailors  pay  only  the  costs  associated  with  college 
coursework  educational  materials.  All  undergraduate  courses  are  from  institutions  that 
are  accredited  by  an  accrediting  body  recognized  by  the  Department  of  Education  and 
affiliated  with  the  Servicemembers  Opportunity  College  degree  program  for  the  Navy, 
ensuring  sailors  the  opportunity  to  transfer  credits  and  complete  degrees  (OPNAVINST 
1560.9A,  2008). 

Over  time,  numerous  Navy  administrative  messages  (NAV ADMINs)  have 
emphasized  the  importance  of  an  educated  Navy  force,  highlighted  the  Navy’s 
commitment  to  providing  educational  opportunities  for  sailors,  and  stressed  the  difficulty 
of  successfully  completing  courses  and  degrees  alongside  performing  official  Navy 
duties.  This  is  well  illustrated  by  the  Navy’s  decision  in  2005  to  require  an  associate’s 


1 


degree  for  E8  selection  in  fiscal  year  2011  and  its  subsequent  decision  to  rescind  this 
requirement  in  2007  (NAVADMIN  203/05,  2005;  NAVADMIN  150/07,  2007). 

Several  of  these  messages  specifically  addressed  changes  to  NCPACE  policies 
and  guidelines.  In  May  2004,  NAVADMIN  109/04  announced  an  interim  NCPACE 
policy  that  aimed  to  make  the  program  more  cost-effective  and  yield  higher  success  rates 
among  participants.  The  policy  announced  that  participants  were  required  to  receive 
written  approval  from  their  commands  and  limited  enrollment  to  two  courses  per  tenn, 
with  first  time  Technology  PACE  students  limited  to  one  course.  The  message  stipulated 
minimum  course  enrollment  levels,  stipulated  ratios  of  computers  to  students,  and  called 
for  command  leadership  to  play  a  more  active  role  in  ensuring  NCPACE  successes. 

In  June  2007,  NAVADMIN  161/07  announced  additional  provisions  for  sailors 
enrolling  in  NCPACE  courses.  This  message  stated  that  participants  were  required  to 
have  passed  their  most  recent  advancement  exam  (or  have  approval  from  their 
commanding  officer),  disallowed  enlisted  members  with  fewer  than  twenty  years  of 
service  who  had  less  than  one  year  remaining  on  their  agreement,  and  required  education 
plans  for  sailors  who  have  completed  their  fifth  course. 

In  February  2008,  NAVADMIN  042/08  rescinded  the  restriction  for  enlisted 
sailors  with  less  than  one  year  of  service  left  and  reiterated  guidelines  for  command 
involvement  and  the  requirement  for  sailors  who  have  taken  more  than  five  courses  to 
commit  to  their  education  plan.  It  placed  additional  restrictions  against  taking  courses 
towards  degrees  already  earned,  with  exceptions  granted  for  sailors  who  change  their 
designator  or  rating. 

In  March  2010,  NAVADMIN  105/10  added  additional  requirements  for  NCPACE 
participants  and  reemphasized  command  leadership  involvement.  It  required  the 
aforementioned  individual  education  plan  for  all  enrollees,  regardless  of  number  of 
classes  completed,  and  restricted  enrollment  to  classes  that  lead  to  a  degree.  A 
prerequisite  of  completing  one  year  at  the  first  pennanent  duty  station  before  seeking 
approval  for  NCPACE  courses  was  added.  It  directly  addressed  the  issue  of  course 


2 


completions  by  calling  for  those  having  trouble  with  their  coursework  to  talk  to  their 
academic  advisors  and  command  leaders  before  dropping  a  course. 

A  recent  study  indicates  a  significant  rise  in  the  number  and  percentage  of  sailors 
enrolling  in  distance  education  courses  through  the  Navy’s  Tuition  Assistance  (TA) 
program  between  1995  and  2008.  This  growth  in  distance  education  enrollments  has 
been  attributed  to  the  increase  in  operational  tempo  since  September  2001.  Distance 
learning  course  completion  rates  are  reported  as  lagging  behind  traditional  course 
completion  rates  for  all  years  considered,  with  Physical  Science  courses  experiencing  the 
biggest  differences  in  completion  rates  between  instruction  methods  (Woosely,  2009). 

Though  the  study  referenced  above  makes  an  inferential  connection  between 
distance  learning  and  operational  tempo  events,  it  and  other  evaluations  of  military- 
sponsored  continuing  education  programs  done  in  the  past  do  not  directly  address  how 
students  perform  academically  in  operational  environments.  This  thesis  attempts  to  fill 
that  gap  by  evaluating  the  effectiveness  of  the  NCPACE  component  of  VOLED. 
NCPACE  is  a  good  candidate  for  addressing  the  effectiveness  of  distance  education  and 
academic  success  in  operational  environments  in  general,  as  the  program’s  students  are 
all  on  sea  duty  and  the  program  has  a  distance  education  component.  Identifying  the 
factors  associated  with  successful  NCPACE  course  completions  will  provide  the 
information  needed  to  make  VOLED  programs  more  effective  as  a  whole. 

B.  OBJECTIVES 

The  primary  objective  of  this  thesis  is  to  detennine  factors  associated  with 
successful  NCPACE  course  completions.  Both  participant  and  course  characteristics  will 
be  evaluated.  It  will  also  attempt  to  quantify  the  impact  a  failed  course  has  on  the  sailor’s 
desire  to  take  additional  college  courses  and  factor  changes  made  to  NCPACE  policies 
and  guidelines  over  time.  This  thesis  characterizes  what  leads  to  a  sailor’s  ability  to 
succeed  in  continuing  education  programs  in  operational  environments  and  to  serve  as  an 
aid  in  screening  and  counseling  VOLED  participants.  This  thesis  also  looks  for  trends  in 
program  participation  rates  and  passing  rates  over  time.  The  course  completion  analysis 
will  be  done  at  the  sailor  level  with  respect  to  the  first  course  taken. 
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The  secondary  objective  of  this  thesis  is  to  determine  if  the  outcomes  of  NCPACE 
courses  have  an  effect  on  promotion  and  retention. 

1.  Literature  Review 

a.  Garcia  and  Joy  (1 998) 

Prior  analyses  of  DoD  VOLED  programs  focused  on  Tuition  Assistance 
(TA)  programs  and  were  primarily  concerned  with  how  TA  usage  impacted  promotion 
and  retention.  Garcia  and  Joy  (1998)  published  one  of  the  few  studies  that  specifically 
identified  factors  associated  with  succeeding  in  NCPACE  in  addition  to  addressing  the 
effectiveness  of  Navy  VOLED  as  a  whole.  Their  model  distinguished  the  following 
sailor  and  course  characteristics  as  factors  that  caused  significant  differences  in  NCPACE 
college  level  course  completion  rates: 

•  Higher-ranked  sailors  had  higher  completion  rates:  71  percent  of 
courses  taken  by  E 1  and  E2s  were  successfully  completed,  whereas  8 1 
percent  of  courses  taken  by  E3  through  E9s  were  successfully 
completed. 

•  Instructor-based  classes  had  higher  completion  rates:  courses  taken 
through  Technology  PACE  had  a  lower  completion  rate  than  those 
taken  through  Instructor  PACE  (77  percent  of  Technology-based 
classes  were  successfully  completed  versus  84  percent  of  Instructor- 
based  classes). 

•  Mathematics  classes  had  lower  completion  rates:  67  percent  of 
mathematics  classes  were  successfully  completed,  while  8 1  percent  of 
non-mathematics  classes  were  successfully  completed. 

•  First  attempts  are  more  successful  than  second  attempts:  80  percent  of 
first  courses  were  successfully  completed,  whereas  only  67  percent  of 
second  courses  were  successfully  completed. 


4 


•  Submariners  had  higher  completion  rates:  completion  rates  aboard 
submarines  were  at  least  10  percentage  points  higher  than  completion 
rates  aboard  other  ship  types. 

•  Orientation  participation,  AFQT  score,  and  previous  college 
experience  were  also  associated  as  factors  that  affected  completion 
rates. 

•  For  lower  division  courses,  NCPACE  success  rates  were  lower  than 
success  rates  associated  with  TA:  Garcia  and  Joy  (1998)  hypothesized 
that  completion  rates  were  lower  for  NCPACE  courses  because  TA 
users  were  required  to  pay  back  any  assistance  used  towards  a  failed 
course  and  were  typically  in  less  demanding  work  environments. 

These  completion  rate  estimates  were  based  on  NCPACE  course  data 
dating  between  July  1995  and  May  1996.  Garcia  and  Joy  (1998)  did  the  regression 
analysis  in  two  stages  to  correct  for  a  potential  selection  bias  that  may  exist  between 
Instructor  PACE  and  Technology  PACE  courses,  as  Technology  courses  have 
characteristics  that  may  make  them  more  demanding.  The  first  regression  modeled  the 
probability  of  receiving  instruction  electronically.  The  second  regression  modeled  the 
probability  of  successfully  completing  a  PACE  course  using  a  correction  factor  derived 
from  the  first  model. 

Both  the  promotion  and  retention  regression  models  were  estimated  in  two 
stages  in  the  same  manner  as  the  NCPACE  model.  This  was  done  to  correct  selection 
biases  in  which  may  exist  as  a  result  of  VOLED  participants  having  higher  than  average 
motivation.  The  first  stage  modeled  VOLED  participation  using  involvement  in 
academic  counseling  as  an  instrumental  variable.  This  was  used  to  calculate  a  correction 
factor  for  the  promotion  and  retention  models.  Other  variables  considered  in  the 
promotion  and  retention  models  were  AFQT  score,  sea  experience,  age,  marital  status, 
and  rating  group. 

A  positive  relationship  was  found  between  college  credits  earned  through 
VOLED  as  a  whole  and  promotion  to  E5  in  five  years.  Promotion  estimates  were  based 
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on  a  single  group  of  active  duty  enlisted  sailors  who  accessed  in  the  last  two  quarters  of 
fiscal  year  1992  and  served  in  the  Navy  at  least  five  years.  The  ordinal  nature  of 
paygrade  was  captured  by  using  an  ordered  probit  model.  The  extent  of  vacancies  in  the 
rating  was  also  considered  in  the  promotion  model. 

Regression  estimates  of  retention  were  based  on  a  single  fiscal  year  1992 
cohort  of  active  duty  enlisted  sailors  with  four-year  contracts.  A  positive  relationship 
was  found  between  college  credits  earned  through  VOLED  and  reenlistments. 
Reenlistment  bonus  information  was  also  captured  in  the  retention  model.  One  of  the 
shortcomings  of  this  analysis  is  that  less  than  a  year’s  worth  of  NCPACE  course  data  was 
used  to  evaluate  the  effectiveness  of  that  component  and  only  a  single  cohort  was 
evaluated  for  the  entire  VOLED  program.  In  addition,  the  two-stage  estimation  methods 
in  this  study  have  limitations  because  the  instrumental  variable  chosen  (enrolling  in 
distance  learning  courses,  participating  in  academic  counseling)  and  the  respective 
outcome  (successfully  completing  a  course,  promoting  to  E5  in  five  years)  is  likely 
correlated  with  the  unobservable  factor  of  individual  motivation. 

b.  Buddin  and  Kapur  (2002) 

Buddin  and  Kapur  estimated  the  effects  of  TA  participation  on  first-term 
Navy  and  Marine  Corps  reenlistments.  It  criticized  the  retention  model  in  Garcia  and  Joy 
(1998)  for  including  sailors  who  separate  from  the  Navy  before  the  end  of  their  term,  as 
these  sailors  do  not  have  the  same  chance  to  utilize  the  TA  program  as  sailors  who  stay 
the  length  of  their  contract.  They  found  that  when  members  who  separated  early  were 
excluded  from  the  sample,  TA  users  were  less  likely  to  reenlist  in  the  Navy  by  9 
percentage  points. 

This  study  did  not  factor  TA  program  performance,  and  was  limited  to 
sailors  and  marines  who  completed  their  first  tenn  in  fiscal  year  (FY)  1997  and  the  first 
half  of  FY  1998. 
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c. 


Sticha  et  al.  (2003) 


This  study  evaluated  the  Army  Continuing  Education  System  (ACES). 
Similar  to  the  Garcia  and  Joy  study,  reenlistment  and  promotion  of  first-term  Active  Duty 
Army  enlisted  soldiers  were  used  as  measures  of  effectiveness. 

Sticha  et  al.  estimated  that  TA  participation  resulted  in  a  7.6  percent 
increase  in  reenlistment  rates.  The  reenlistment  analysis  looked  at  soldiers  who  began 
and  completed  a  three-  or  four-year  contract  between  October  1995  and  September  2001 
and  completed  enough  service  to  become  eligible  for  reenlistment.  Soldiers  stationed  at 
installations  where  the  Educational  Management  Infonnation  System  (EDMIS)  database 
was  not  operational  were  excluded.  The  authors  acknowledged  this  resulted  in  a  biased 
sample,  as  those  stationed  at  bases  in  which  EDMIS  is  operational  do  not  make  up  a 
random  sample  of  all  first-time  enlistees.  Factors  considered  in  the  reenlistment  model 
were  paygrade,  military  specialty,  reenlistment  bonus  information,  Career  Management 
Field  (CMF),  race,  sex,  level  of  education  and  marital  status  in  addition  to  participation  in 
TA  and  Functional  Academic  Skills  Training  (FAST)  components  of  ACES.  A  bivariate 
probit  model  which  simultaneously  estimated  participation  in  TA  and  reenlistment  was 
used  to  control  for  selection  bias. 

Performance  and  promotion  analysis  was  based  on  data  from  the  NC021 
Validation  Project,  which  collected  information  about  ACES  participation  from  Non¬ 
commissioned  officers  (NCOs).  Sticha  et  al.  found  that  participation  in  TA,  MOSIT 
(Military  Occupation  Specialty  Improvement  Training)  courses,  and  NCO  Leader  Skill 
Enhancement  courses  are  positively  associated  with  promotion  and  performance.  One  of 
the  shortcomings  of  the  database  used  for  the  performance  and  promotion  analysis  is  that 
it  had  fewer  than  2,000  soldiers  and  the  course  data  relied  on  self-reported  information. 

d.  Mehay  and  Pema  (2008) 

Mehay  and  Pema  analyzed  the  impact  of  the  Navy’s  TA  program  on  the 
retention  and  promotion  of  first-term  Navy  enlisted  personnel.  Their  approach  to  the 
selection  bias  problem  differs  from  the  two  previously  mentioned  studies  in  that  they 
used  a  sample  consisting  solely  of  TA  participants,  assuming  that  sailors  who  choose  to 
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participate  in  TA  have  similar  motivations  and  that  those  who  withdraw  were  forced  to 
do  so  by  external  circumstances  such  as  military  deployments  and  job  duties.  Their 
choices  of  control  variables  in  their  retention  and  promotion  models  were  similar  to  those 
in  previous  studies:  they  used  race,  ethnicity,  sex,  education,  AFQT  scores,  marital 
status,  and  dependents.  Like  the  ACES  study,  Mehay  and  Pema  considered  multiple 
cohorts  instead  of  a  single  one. 

They  noted  that  women  used  TA  at  twice  the  rate  of  men  and  had  higher 
rates  of  successful  course  completion.  They  also  found  that  minority  groups  were  more 
likely  to  use  TA  than  whites.  Their  study  found  that  sailors  who  use  the  TA  program  to 
enroll  in  college  classes  have  a  significantly  higher  probability  of  reenlistment  and  of 
promotion  to  both  E4  and  E5  than  those  who  participate  but  do  not  complete  their 
courses.  They  ran  models  with  the  full  sample  and  compared  them  to  the  models  that 
included  only  TA  participants  and  estimated  the  self-selection  bias  difference  between 
the  full  sample  and  the  TA  participants.  One  potential  problem  is  the  assumption  that 
sailors  withdraw  due  to  external  circumstances,  which  may  be  too  strong.  The  descriptive 
statistics  for  the  restricted  sample  demonstrate  that  course  completers  have  higher  AFQT 
scores  and  also  indicate  that  that  sailors  who  attained  higher  levels  of  education  had  more 
success  in  completing  courses. 

C.  SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

This  study  is  restricted  to  Navy  enlisted  personnel  who  participated  in  the 
NCPACE  program  between  fiscal  years  1995  and  2008.  We  excluded  warrant  and 
commissioned  officers  from  the  statistical  models  in  order  to  obtain  a  more  homogenous 
sample. 

NCPACE  courses  were  only  available  to  commands  with  a  sea  duty  unit 
identification  code  (UIC)  of  type  4,  some  remote  locations,  and  to  sailors  supporting 
contingency  operations  (OPNAVINST  1560.9A).  Sailors  were  also  subjected  to 
additional  requirements,  which  evolved  over  time.  Because  data  that  determines  a 
sailor’s  eligibility  to  participate  in  NCPACE  at  a  given  time  was  not  available,  we  did  not 
consider  program  participation  rates. 
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The  promotion,  extension,  and  reenlistment  models  are  limited  in  the  sense  that 
we  did  not  obtain  data  which  determines  a  sailor’s  eligibility  to  promote  (such  as 
promotion  board  data)  or  motivators  for  extension  and  reenlistment  decisions  (such  as 
benefits  or  bonuses  for  these  actions). 

D.  COURSE  OF  STUDY 

Chapter  II  provides  a  description  of  the  data  sources  and  sets,  some  descriptive 
statistics  and  univariate  analyses  of  course  completion  rate  by  variables  of  interest. 
Chapter  III  provides  the  variables,  assumptions  and  results  of  the  course  completion, 
effects  of  failed  courses,  promotion,  and  reenlistment  models.  Chapter  IV  provides 
recommendations  and  conclusions.  A  list  of  references  is  provided  at  the  end. 
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II.  DATA  AND  DESCRIPTIVE  STATISTICS 


A.  DATA  SOURCES  AND  SETS 

1.  Navy  College  Management  Information  System  (NCMIS) 

NCMIS  was  the  source  of  NCPACE  participation  data.  This  data  set  is  an 
accumulation  of  course  rosters  for  NCPACE  classes  administered  between  fiscal  years 
1995  and  2009.  These  data  provide  extensive  information  on  both  students  and  courses. 
More  specifically,  the  NCMIS  data  set: 

•  identifies  course  enrollments  and  results  by  individual  sailor,  making  it 
possible  to  track  individuals  who  have  taken  more  than  one  course; 

•  provides  demographic  information  about  the  sailor  (race,  ethnicity,  sex), 
as  well  as  characteristics  about  him  or  her  (rank,  rating,  education  level, 
AFQT  score)  at  the  time  the  request  to  take  the  course  was  made; 

•  describes  course  attributes,  such  as  the  title,  level  (lower  division,  upper 
division,  graduate),  credit  hours  and  delivery  method; 

•  gives  infonnation  about  the  school,  the  command,  the  NCPACE  office 
that  sponsored  the  course,  the  fiscal  year  that  the  course  request  was 
made,  and  some  information  about  the  associated  contract  (contract 
number  and  costs). 

The  course  start  date  was  used  to  determine  the  fiscal  year  the  course  was  taken. 
Records  corresponding  to  courses  beginning  in  fiscal  year  2009  were  removed  from  our 
sample  because  the  set  of  fiscal  year  2009  courses  appeared  to  be  incomplete. 

2.  Defense  Manpower  Data  Center  (DMDC) 

DMDC  provided  sailor-level  infonnation  on  Active  Duty  accessions,  strength, 
reenlistments  and  separations  between  October  1994  and  December  2009. 

The  accession  data  set  contained  accession  dates,  AFQT  score  percentile,  and  a 
prior  military  service  indicator  at  the  time  of  accession. 
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The  strength  data  set  provided  quarterly  snapshots  of  the  Active  Duty  force  and 
included  the  following:  branch  of  service,  paygrade  and  date  of  rank,  unit  identification 
code,  time  in  service,  primary  occupation,  sex,  race,  ethnicity,  marital  status,  current 
enlistment  date,  education  level  and  enlistment  contract  length.. 

The  reenlistment  and  separation  data  set  contains  the  date  of  the  action.  The 
separation  data  contains  separation  codes  in  addition  to  the  action  date. 

B.  DESCRIPTIVE  STATISTICS 

1.  Overall  NCPACE  Course  Enrollments 

NCMIS  reported  262,479  NCPACE  course  records  between  fiscal  year  1995  and 
fiscal  year  2009.  This  section  considers  260,803  records,  which  corresponded  to  courses 
with  valid  course  start  dates  between  fiscal  year  1995  and  fiscal  year  2008. 

The  number  of  overall  NCPACE  courses  taken  varied  by  fiscal  year,  with  an 
increase  between  1995  and  2001,  a  decrease  between  2001  and  2005,  and  an  increase 
between  2005  and  2006  (Figure  1).  The  proportion  of  Technology  courses  taken  varied 
between  32.8  percent  (fiscal  year  2004)  and  51.8  percent  (fiscal  year  2007).  The 
decrease  in  overall  participation  between  2001  and  2005  may  be  explained  by  increases 
in  sailor  duties  as  a  result  of  the  events  that  occurred  on  September  2001.  It  is  likely  not 
explained  by  changes  in  the  number  of  eligible  participants,  as  the  changes  in  the  number 
of  courses  taken  did  not  correspond  to  changes  in  Navy  enlisted  end  strength  figures 
during  this  time  period  (Figure  2). 


12 


0 

1995 

1996 

1997 

1998 

1999 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

■  Technology 

215 

6948 

7076 

6185 

8747 

12196 

10613 

6552 

4212 

2524 

9612 

12348 

12568 

8970 

□  Instructor 

220 

8292 

11858 

10472 

14422 

16645 

19304 

10947 

6775 

5171 

12187 

12854 

11699 

11191 

Fiscal  Year 


Figure  1.  NCPACE  Courses  Taken  by  Fiscal  Year  (FY1995-FY2008). 

Source:  NCMIS. 


Figure  2.  Navy  Enlisted  End  Strength  by  Fiscal  Year  (FY1995-FY2008). 

Source:  DMDC. 


The  proportion  of  successful  course  completions  generally  hovered  around  70 
percent,  with  a  decrease  in  completion  rates  that  began  in  1999  that  lasted  through  2003 
(Figure  3;  note  that  the  vertical  scale  starts  at  50  percent).1  Excluding  Invalid/Unknown 


1  Succeeded  was  defined  as  attaining  a  course  grade  of  A,  B,  B+,  C,  or  S.  Did  Not  Succeed  was 
defined  as  attaining  a  course  grade  of  D,  DP,  F,  FN,  I,  N,  NC,  NF,  NS,  W,  WA,  WD,  WF,  WP  or  WX. 
Invalid/Unknown  course  grades  were  not  factored  into  the  denominator  of  the  percentages  but  are  shown  in 
the  graph. 
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observations,  the  highest  course  completion  rate  was  experienced  in  2008;  however,  this 
year  also  experienced  the  highest  number  of  invalid  and  unknown  course  grades. 
Changes  in  course  completion  rates  could  be  the  result  of  NAV ADMIN  109/04, 
NAVADMIN  161/07  and  NAVADMIN  042/08. 


Figure  3.  NCPACE  Success  Rate  by  Fiscal  Year  (FY1995-FY2008). 

Source:  NCMIS. 


An  overwhelming  majority  of  NCPACE  courses  were  taken  by  enlisted 
personnel.  Between  fiscal  years  1995  and  2008,  97  percent  of  course  attempts  were  made 
by  enlisted  personnel,  with  82.5  percent  of  attempts  made  by  sailors  with  pay  grades  of  E3 
through  E6  (Figure  4).  The  remainder  of  this  thesis  is  restricted  to  Navy  enlisted 
personnel,  with  the  assumption  that  this  population  is  homogenous  in  motivation. 


Figure  4.  Courses  taken  by  Pay  Grade  at  the  Time  of  the  Course 
(FY1995-FY2008).  Source:  NCMIS. 
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2. 


First-time  NCPACE  Enrollments 


A  considerable  number  of  participants  have  taken  more  than  one  course.  Figure  5 
shows  the  number  of  courses  taken  versus  the  number  of  participants  by  fiscal  year.  As  a 
key  objective  of  this  thesis  is  to  serve  as  a  tool  for  screening  participants,  we  have  chosen 
to  evaluate  the  first  course  taken  for  each  participant.  We  assume  that  participants  who 
make  second  attempts  and  beyond  can  be  screened  based  on  previous  performance. 


Figure  5.  NCPACE  Courses  versus  Students  by  Fiscal  Year  (FY1995-FY2008). 

Source:  NCMIS. 

A  data  set  containing  first-time  NCPACE  enrollees  with  valid  enlisted  paygrades 
was  extracted  from  the  NCMIS  course  roster.  We  chose  the  record  with  the  earliest  start 
date  for  each  individual  enrollee  and  validated  that  enrollment  record  against  the 
corresponding  DMDC  quarterly  strength  file.  From  this  data  set,  only  records  with  valid 
course  grades  were  retained.  The  resulting  data  set  contains  1 12,914  observations.  First 
course  attempts  and  success  rates  experience  trends  similar  to  those  of  overall  NCPACE 
course  enrollment  levels  and  success  rates.  The  remainder  of  this  section  examines  how 
first  course  success  rates  vary  by  course  and  sailor  factors  considered  individually. 

The  rate  of  successful  first  course  completions  varied  by  course  attributes.  Three- 
fourths  (75.9  percent)  of  Instructor-based  course  participants  succeeded,  whereas  only 
about  half  (48.1  percent)  of  Technology-based  course  participants  succeeded  (Figure  6). 

Different  success  rates  were  also  observed  across  course  subjects  (Figure  7).  Those  who 
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took  mathematics  (58.3  percent)  and  physical  and  other  science  (50.3  percent)  courses 
did  not  succeed  at  rates  comparable  to  those  who  took  humanities,  social  science  and 
business  courses  (67.1  percent).  In  addition,  the  success  rate  varied  across  the  schools 
that  administered  the  course  (Figure  8).  75.4  percent  of  first  time  participants  who  took 
courses  through  Central  Texas  College  succeeded,  while  only  37.6  percent  of  participants 
who  took  courses  through  Richland  College  succeeded. 
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Figure  6.  NCPACE  First  Course  Success  Rate  by  Course  Delivery  Method 

(FY1995-FY2008).  Source:  NCMIS. 
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Figure  7.  NCPACE  First  Course  Success  Rate  by  Course  Subject  (FY1995-FY2008). 

Source:  NCMIS.2 
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Figure  8.  NCPACE  First  Course  Success  Rate  by  School  (FY1995-FY2008). 

Source:  NCMIS. 


The  rate  of  successful  first  course  completions  also  varied  by  attributes  of  the 
sailor  taking  the  course.  Sailors  who  completed  at  least  some  college  were  most 
successful,  whereas  those  who  held  an  alternative  high  school  credential  were  least 


2  Course  categories  were  determined  by  a  combination  of  course  subject  and  title  reported  by  NCMIS. 
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successful  (Figure  9).  Higher  ranked  sailors  succeeded  at  higher  rates  than  lower  ranked 
sailors  (Figure  10).  Also,  there  is  some  variance  in  success  by  rating  (Figure  1 1)  and  unit 
type  (Figure  12). 
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Figure  9.  NCPACE  First  Course  Success  Rate  by  Education  Level  (FY1995-FY2008). 

Source:  NCMIS. 
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Figure  10.  NCPACE  First  Course  Success  Rate  by  Rank  (FY1995-FY2008). 

Source:  NCMIS. 
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Figure  11.  NCPACE  First  Course  Success  Rate  by  Rating  Group  (FY1995-FY2008). 
Source:  NCMIS  and  Navy  Enlisted  Manpower  and  Personnel  Classifications  and 
Occupational  Standards  ( Navy  Enlisted  Manpower  and  Personnel  Classifications  and 

Occupational  Standards,  20 10). 3 4 


Figure  12.  NCPACE  First  Course  Success  Rate  by  Unit  Type  (FY1995-FY2008). 
Source:  NCMIS  and  Naval  Vessel  Register  (NVR).4 


3  Chapter  III.A.2  describes  the  categorization  of  ratings  in  detail. 

4  Chapter  III.A.2  describes  the  categorization  of  unit  types  in  detail. 
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Some  variation  in  success  rates  were  observed  between  the  following  groups: 

•  AFQT  Category  I-IIIA  (66.3  percent)  and  AFQT  Category  IIIB-V  (61.3 
percent  success  rate). 

•  Married  (68.7  percent)  and  unmarried  (62.5  percent)  sailors. 

Little  variation  in  success  rates  were  observed  in  the  following  sailor 
characteristics: 

•  Sex:  Females  (65.3  percent)  and  males  (64.9  percent). 

•  Race:  American  Indian/Alaskan  Natives  (64.1  percent),  Asians  (66.8 
percent),  Blacks  (64  percent),  Pacific  Islanders  (66.9  percent),  Whites 
(65.9  percent)  and  Other  (63.3  percent). 

•  Ethnicity:  Hispanics  (65  percent)  and  non-Hispanics  (64.3  percent). 

3.  Impact  of  Failing  the  First  Course 

This  study  also  aims  to  quantify  the  impact  of  failing  the  first  course.  We  merged 
information  about  subsequent  NCPACE  course  attempts  to  the  data  set  of  first-time 
enrollments  described  in  the  previous  section  in  order  to: 

•  determine  if  sailors  who  did  not  succeed  at  their  first  attempt  make 
subsequent  attempts. 

•  evaluate  the  perfonnance  of  sailors  who  did  not  succeed  at  their  first 
attempt  who  make  subsequent  attempts. 

For  each  participant,  we  defined  the  subsequent  attempt  as  the  course  with  the 
earliest  reported  start  date  occurring  after  the  first  course’s  reported  completion  date. 
Records  with  second  courses  that  did  not  have  valid  enlisted  paygrades  or  course  grades 
were  not  considered.  This  resulting  data  set  contained  1 12,444  observations,  with  50,474 
instances  of  subsequent  course  attempts. 

Successful  completions  of  first  courses  seem  to  be  positively  associated  with 
enrollments  in  subsequent  courses.  Overall,  44.9  percent  of  NCPACE  participants  made 
a  subsequent  attempt  after  the  completion  of  their  first  course.  Less  than  one -third  (32.1 
percent)  of  the  participants  who  did  not  successfully  complete  their  first  course  made  a 
subsequent  attempt,  whereas  about  half  (51.8  percent)  of  those  who  succeeded  made  a 
subsequent  attempt  (Figure  13). 
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Figure  13.  Rate  of  Subsequent  NCPACE  Course  Attempts  by  Result  of  First  Course 

(FY1995-FY2008).  Source:  NCMIS. 

There  is  also  some  indication  that  successfully  completing  a  first  course  is 
associated  with  successfully  completing  subsequent  ones.  While  the  overall  subsequent 
course  completion  rate  of  66.9  percent  is  comparable  to  the  first  course  completion  rate 
of  65  percent,  there  is  a  noticeable  difference  by  first  course  results.  Seventy-five  percent 
of  participants  who  successfully  completed  their  first  course  and  taken  a  subsequent 
course  passed  the  subsequent  course,  whereas  only  42  percent  of  participants  who  did  not 
successfully  complete  their  first  course  and  taken  a  subsequent  course  passed  the 
subsequent  course  (Figure  14). 
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Figure  14.  Success  Rate  of  Subsequent  NCPACE  Courses  by  Result  of  First  Course 

(FY1995-FY2008).  Source:  NCMIS. 
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4.  Impact  on  Promotion  and  Retention 

Lastly,  this  study  examines  if  NCPACE  course  outcomes  have  promotion  and 
retention  effects  on  first-term  sailors  with  four-year  contracts  and  non-prior  service 
accessions  occurring  between  fiscal  years  1995  and  2004.  We  created  the  following  data 
sets  using  the  source  data: 

•  Using  the  NCMIS  course  roster,  we  consolidated  each  participant’s  course 
history  into  a  single  record,  which  reported  the  course  start  dates 
associated  with  the  first  successful  and  unsuccessful  attempts.  We  also 
captured  NCPACE  participant  demographics  at  the  time  of  the  first  course 
to  include  the  fiscal  year  of  the  first  course  attempt. 

•  Using  the  DMDC  accession  data,  we  identified  non-prior  service  Navy 
accessions,  capturing  the  date  of  accession. 

•  Using  the  DMDC  strength  data,  we  captured  additional  NCPACE 
participant  demographic  infonnation  at  the  time  of  the  first  course  and 
identified  participants: 

o  with  48-month  contracts. 

o  who  continued  to  serve  beyond  their  48-month  contract. 

o  who  promoted  to  E5  before  the  end  of  their  48-month  contract. 
We  chose  to  evaluate  promotions  to  E5  due  to  the  competitive 
nature  of  receiving  this  promotion  within  the  first  four  years  of 
service. 

•  Using  the  DMDC  reenlistment  and  separation  data,  we  captured  the  first 
reenlistment  or  separation  decision. 

These  datasets  were  merged.  In  a  manner  analogous  to  that  in  the  study  of  Mehay 
and  Pema  (2008),  we  restricted  our  sample  to  NCPACE  participants  who  took  at  least 
one  course  during  their  first  term,  with  a  successful  participant  defined  as  a  sailor  who 


22 


successfully  completed  at  least  one  course  in  their  first  term.  We  defined  the  first  term  as 
the  period  between  accession  and  48  months  thereafter.5  The  resulting  data  set  contains 
47,406  observations.6 


We  found  that  those  who  succeeded  in  at  least  one  NCPACE  course  promoted  to 
E5  within  48  months  of  service  at  higher  rates  than  those  who  did  not  (54.2  versus  49.1 
percent;  Figure  15).  However,  it  is  worth  noting  that  other  things  such  as  rating 
vacancies  are  a  major  factor  of  a  participant’s  promotion  potential.  We  found  that 
extension  and  reenlistment  rates  do  not  vary  much  with  respect  to  NCPACE  success 
(Figures  16  and  17). 
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Figure  15.  Rate  of  Promotion  to  E5  by  NCPACE  Success  (FY1995-FY2003  Accessions). 

Source:  NCMIS  and  DMDC. 


5  We  defined  die  first  term  in  this  manner  versus  using  the  reported  reenlistment  and  separation  action 
dates  because  over  a  quarter  of  the  sample  had  action  dates  that  occurred  more  than  one  year  after  the  end 
of  their  contract  obligation. 

6  We  included  16,686  sailors  who  left  before  serving  36  months  of  their  48-month  contract.  It  is  worth 
noting  that  this  dataset  contains  124  observations  which  report  the  sailor’s  pay  grade  as  E6  or  above  at  the 
time  of  his  or  her  first  course. 
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Figure  16.  Rate  of  Extension  by  NCPACE  Success  (FY1995-FY2003  Accessions). 

Source:  NCMIS  and  DMDC. 
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Figure  17.  Rate  of  Reenlistment  by  NCPACE  Success  (FY1995-FY2003  Accessions). 

Source:  NCMIS  and  DMDC. 
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III.  REGRESSION  ANALYSIS 


A.  COURSE  COMPLETION  MODEL 

The  course  completion  model  seeks  to  identify  sailor  and  course  factors 
associated  with  successful  first  attempts.  We  used  the  data  set  of  first-time  NCPACE 
enrollments  described  in  the  previous  chapter  for  the  course  completion  model. 

1.  Dependent  Variable 

The  dependent  variable  for  this  model  is  an  indicator  variable  that  equals  1  if  the 
course  was  completed  successfully  and  0  if  the  course  was  not  completed  successfully. 
Letter  grades  below  “C  minus,”  non-passing  marks  in  non-letter  graded  courses, 
incompletes,  and  withdrawals  were  defined  as  unsuccessful  attempts.  A  successful 
course  completion  is  defined  as  receiving  a  “C  minus”  or  above  in  letter  graded  courses 
or  a  passing  mark  in  non-letter  graded  courses.  Course  records  with  invalid  grades  were 
dropped  from  the  data  set. 

2.  Independent  Variables 

The  independent  variables  we  chose  are  heavily  based  on  previous  voluntary 
education  studies.  We  considered  the  following  course  characteristics: 

•  Course  delivery  method.  Garcia  and  Joy  (1998)  and  Woosely  (2009) 
reported  differences  between  traditional  classroom  (Instructor)  and 
distance  learning  (Technology)  methods.  Course  delivery  methods  are 
categorized  as  Instructor  or  Technology. 

•  Course  subject.  Garcia  and  Joy  (1998)  and  Woosely  (2009)  indicate  that 
completion  rates  for  mathematics  and  physical  sciences  were  lower  than 
other  courses.  We  used  course  subjects  and  titles  to  categorize  courses 
into  the  following  academic  disciplines:  Humanities  and  Social  Science, 
Business,  Mathematics,  Physical  Sciences,  and  Other  Sciences. 
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•  Unit  vessel  type.  The  study  of  Garcia  and  Joy  (1998)  indicated  that 
submariners  experienced  higher  completion  rates.  We  mapped  hull 
numbers  found  in  Unit  Identification  Code  titles  provided  by  NCMIS  to 
vessel  classification  information  from  the  Naval  Vessel  Register  and 
created  the  following  unit  types: 

o  Surface  Combat 

■  Aircraft  Carrier 

■  Amphibious  Warfare 

■  Coastal  Defense 

■  Mine  Warfare 

■  Surface  Combatant 
o  Submarine 

o  Logistics  and  Support 

■  Combat  Logistics 

■  Mobile  Logistics 

■  Support 
o  Other  Unit  Type 

•  Fiscal  year  of  course  start  date.  This  was  included  in  order  to  control  for 
changes  in  NCPACE  and  Navy  policies  over  time. 

•  School.  Unlike  Tuition  Assistance,  which  can  be  applied  towards  the 
educational  institution  of  the  sailor’s  choice,  NCPACE  courses  are  offered 
by  a  more  or  less  fixed  set  of  schools.  Of  the  19  schools  that  administered 
NCPACE  courses,  three  schools  administered  over  90  percent  of  the 
courses  associated  with  first  attempts.  We  have  categorized  schools  as 
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follows  to  control  for  the  variation  that  may  occur  among  them:  Central 
Texas  College,  Coastline  Community  College,  Richland  College,  and 
Other  School. 

We  considered  the  following  sailor  characteristics  in  the  course  completion 
model.  Factors  that  may  vary  over  time  were  captured  around  the  start  of  the  course: 

•  AFQT  percentile  score.  AFQT  percentile  score  is  a  continuous  variable 
ranging  from  0  to  99  which  captures  the  sailor’s  performance  on  the 
Armed  Forces  Qualification  Test. 

•  Education.  Education  reports  the  level  of  education  attained  by  the  sailor 
and  has  been  used  in  past  studies  to  capture  an  individual’s  motivation. 
We  have  chosen  the  following  categories:  Did  Not  Complete  High  School, 
received  an  Alternative  High  School  Credential,  received  a  High  School 
Diploma,  and  Completed  at  Least  Some  College. 

•  Rank.  Enlisted  ranks  are  treated  as  categorical  variables. 

•  Rating.  Navy  enlisted  ratings  are  classified  into  23  occupational  fields  as 
defined  by  Navy  Enlisted  Manpower  and  Personnel  Classifications  and 
Occupational  Standards.  We  grouped  the  occupational  fields  into  five 
areas  based  on  functional  area: 

o  Administrative 

■  Administration  (LN,  NC,  PS,  RP,  YN) 

■  Communications  (IT) 

■  Logistics  (LS,  SH) 

■  Media  (MC) 

■  Music  (MU) 
o  First  Responder 

■  Health  Care  (HM) 
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Law  Enforcement/Security  (MA) 


o  Intelligence  and  Special  Warfare 

■  Cryptology  (CTI,  CTM,  CTN,  CTR,  CTT) 

■  Intelligence  (IS) 

■  Special  Warfare/Diving  (EOD,  ND,  SB,  SO) 
o  Maintenance 

■  Aviation  Ground  Support  (ABE,  ABF,  ABH,  AS) 

■  Aviation  Maintenance/Weapons  (AD,  AE,  AM,  AME,  AO, 
AT,  AZ,  PR) 

■  Construction  (BU,  CE,  CM,  EA,  EO,  SW,  UT) 

■  General  Seamanship  (BM) 

■  Marine  Engineering  (EM,  EN,  GSE,  GSM,  IC,  MM) 

■  Ordnance  Systems  (GM,  MN,  MT) 

■  Ship  Maintenance  (DC,  HT,  MR) 

■  Weapons  Control  (ET,  FC,  FT) 
o  Operations 

■  Air  Traffic  Control  (AC) 

■  Aviation  Sensor  Operations  (AW) 

■  Meteorology  and  Oceanography  (AG) 

■  Sensor  Operations  (STG,  STS) 

■  Ship  Operations  (OS,  QM) 

•  Race.  We  grouped  the  races  into  the  following  categories:  White, 
American  Indian/Alaska  Native,  Asian,  Black  or  African  American, 
Native  Hawaiian  or  other  Pacific  Islander,  and  Other  Race. 


28 


•  Ethnicity.  We  divided  this  group  into  Hispanic  and  non-Hispanic.  Sailors 
with  an  ethnicity  of  Cuban,  Mexican,  Puerto  Rican,  Latin  American  of 
Hispanic  Descent,  and  Other  Hispanic  Descent  were  defined  as  Hispanic. 

•  Sex.  We  divided  this  group  into  Male  and  Female. 

•  Marital  status.  We  divided  this  group  into  Married  and  Not  Married. 

3.  Model  Assumptions,  Model,  and  Model  Diagnostics 

Logistic  regression  models  were  used  to  predict  outcomes  throughout  this  chapter. 
To  predict  the  successful  completion  of  the  first  course  for  participant  i,  we  assume 
Course  completion is  a  Bernoulli  random  variable  with  probability  distribution  as 
follows: 


Course  completion ,■ 

Probability 

1 

P( Course  completion  =  1 )  =  n, 

0 

P( Course  completion,  =  0)  =1  -  n, 

Table  1.  Probability  Distribution  of  Course  completion , 


We  chose  the  following  logistic  response  function: 

9^{XiP) 

E( m)  =  1  + 

where  xj  =  [  1,  Course  delivery  methods  Marital  statusj  ]  and  fV  =  [  [hi,  fh,  ....  [h:i  ]■ 

tr 

The  equivalent  logit  transformation  is  In  1  _  fT  =  [hi  +  [it  Course  delivery  method  +  [hi 
Course  subject t  +  /??,  Unit  typet  +  /C,  Fiscal  yeart  +  [hi  School,  +  [h,  AFQT  +  /i7( 
Education t  +  /?«  Rank,  +  [jg,  Ratingi  +  [iio,  Facet  +  [hi  Ethnicity  +  fi  12  Sex  +  [3 13  Marital 
status  (Montgomery,  2001).  The  analysis  of  deviance  table  corresponding  to  the  course 
completion  model  demonstrates  that  each  of  the  selected  factors  is  statistically  significant 
at  the  .05  level  (Table  2). 
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Df 

Deviance 

Resid.  Df 

Resid.  Dev 

Pr (Chi) 

NULL 

101000 

132000 

Technology 

1 

8330 

101000 

123000 

0 

CourseType 

4 

585 

101000 

123000 

0 

UnitType 

3 

875 

101000 

122000 

0 

FY 

13 

1400 

101000 

120000 

0 

School 

3 

654 

101000 

120000 

0 

Afqt 

1 

1360 

101000 

118000 

0 

Education 

3 

706 

101000 

118000 

0 

Rank 

8 

2570 

101000 

115000 

0 

Rating 

4 

41 . 9 

101000 

115000 

1.76E-08 

Race 

5 

37.3 

101000 

115000 

5.22E-07 

Hispanic 

1 

11.9 

101000 

115000 

0.000567 

Female 

1 

65 

101000 

115000 

7.77E-16 

Married 

1 

19.1 

101000 

115000 

1 .22E-05 

Table  2.  Course  Completion  Model  Analysis  of  Deviance  Table 


We  used  the  Hosmer-Lemeshow  test  to  assess  the  goodness-of-fit  of  the  logistic 
regression  models  in  this  thesis.  This  test  evaluates  subgroups  of  subjects  based  on  the 
percentile  of  the  estimated  probabilities  from  the  model  and  measures  the  difference 
between  the  observed  number  of  occurrences  (from  the  data  set)  and  the  expected  number 
of  occurrences  (based  on  the  model)  for  each  subgroup.  The  test  statistic  is  defined  as: 

y  {\-n'kKkf 

^ raffed  “ 


where  n  subjects  are  divided  into  g  subgroups  each  of  approximate  size  «//  =  nig.  If  g  = 


100,  the  Mi  group  contains  subjects  in  the  Mi  percentile.  Then  we  define  as  ;=1 


•rr 

2> 


*1  „ 
y  fTCf 

Ls~HT~ 

the  observed  number  of  completions  in  the  Mi  group,  and  as  /=i  K  ,  the  expected 

number  of  completions  in  that  group  under  the  model.  The  distribution  of  C  is 
approximated  by  the  chi-square  distribution  with  g  -  2  degrees  of  freedom  (Hosmer  and 
Lemeshow,  1989). 


Because  of  the  size  of  our  data  sets,  we  chose  100  as  the  value  of  g.  The  resulting 
Hosmer-Lemeshow  test  statistic  for  the  course  completion  model  has  a  p-value  of  0.0098 
(Chi-squared  =  133.608,  df  =  98),  and  Figure  18  is  a  plot  of  the  observed  versus  expected 
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values  for  the  course  completion  model.  Although  the  p-value  of  the  test  statistic  is 
slightly  smaller  than  .01,  the  plot  indicates  that  the  model  fits  the  data  “reasonably”  well. 


Observed  vs.  Expected 


0.2  0.4  0.6  0.8 

%  Observed  Per  Group 


Figure  18.  Hosmer-Lemeshow  Plot  of  Course  Completion  Model 

The  Receiver  Operating  Characteristic  (ROC)  is  a  function  that  relates  the  hit  rate 
(sensitivity)  to  the  false-alann  rate  (false  positive  rate).  The  diagonal  line  where 
sensitivity  =  false  positive  rate  represents  the  chance  line,  that  is,  the  ROC  for  a 
classification  which  predicts  completion  purely  at  random  (Egan,  1975).  The  area  under 
a  model’s  ROC  curve  (AUC)  suggests  that  model’s  ability  to  accurately  predict  the  value 
of  an  observation’s  response.  Hosmer  and  Lemeshow’s  guidelines  for  interpreting  AUC 
values  are  in  Table  3  (Minitab  Technical  Support  Document). 


AUC  =  0.5 

No  discrimination 

0.7  <  AUC  <  0.8 

Acceptable  discrimination 

0.8  <  AUC  <  0.9 

Excellent  discrimination 

AUC  >  0.9 

Outstanding  discrimination 

Table  3.  Guidelines  for  Interpreting  Area  Under  the  ROC  Curve.  Source:  Minitab 

Technical  Support  Document. 
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Figure  19  shows  the  ROC  curve  for  the  course  completion  model.  We  have 
“acceptable”  classification  results,  as  all  points  of  the  ROC  are  above  the  chance  line  and 
the  area  under  the  curve  (AUC)  is  0.732. 


ROC 


Figure  19.  ROC  Curve  of  Course  Completion  Model 

4.  Analysis 

The  results  of  the  course  completion  model  are  presented  in  Table  4.  We  reject 
the  null  model  in  favor  of  this  model,  as  our  difference-in-deviance  test  statistic  16646.2 
is  greater  than  x~a,48  for  any  reasonable  choice  for  a.  The  results  of  this  fixed-effects 
model  corroborated  the  following  findings  in  the  study  of  Garcia  and  Joy  (1998): 

•  The  predicted  success  rate  increases  with  rank.  E4s  were  twice  as  likely  to 
successfully  complete  their  first  course  than  Els.  E6s  were  about  three 
times  more  likely  to  complete  than  Els. 

•  Instructor-based  courses  have  higher  predicted  success  rates.  Participants 
who  take  Instructor  PACE  courses  were  about  are  ten  times  more  likely  to 
succeed  than  those  taking  Technology  PACE  courses. 

•  Mathematics  courses  have  lower  predicted  success  rates.  We  found  that 
Science  courses  also  have  lower  predicted  success  rates. 

•  Submarine  units  have  higher  predicted  success  rates. 
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We  also  observed  the  following: 


•  Higher  AFQT  scores  and  levels  of  education  were  associated  with  higher 
predicted  success  rates.  Participants  who  have  taken  some  college  courses 
were  twice  as  likely  to  succeed  compared  to  those  who  did  not  finish  high 
school. 

•  Courses  sponsored  by  Coastline  Community  College  have  higher 
predicted  success  rates. 
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(Intercept) 


Technology 


CourseTypeBus 


CourseTypeMath 


CourseTypePSci 


CourseTypeOSci 


UnitTypeOther 


UnitTypeSubmarine 


UnitTypeSurfaceCombat 


FY1996 


FY1997 


FY1998 


FY1999 


FY2000 


FY2001 


FY2002 


FY2003 


FY2004 


FY2005 


FY2006 


FY2007 


FY2008 


SchoolCoastline 


SchoolRichland 


SchoolOther School 


Afqt 


EducationaltHS 


EducationHS 


EducationsomeColl 


RankE2 


RankE3 


RankE4 


RankE5 


RankE6 


RankE7 


RankE8 


RankE9 


RatingFR 


Ratingln 


RatingMa 


RatingOp 


RaceAIAN 


RaceAsian 


RaceBlack 


RacePI 


RaceOtherRace 


Hispanic 


Female 


Married 


-0.144 


-2.3 


0.0561 


-0.514 


-0.0758 


-0.239 


0.294 


0.642 


0.159 


-0.218 


-0.335 


-0.435 


-0.75 


-0.832 


-0.815 


-0.913 


-0.792 


-0.376 


-0.453 


-0.293 


-0.246 


-0.219 


0.695 


-0.0417 


0.226 


0.0123 


0.0555 


0.301 


0.675 


0.207 


0.354 


0.687 


0.931 


1.22 


1.41 


1 . 64 


1.59 


0.0604 


0.247 


0.0986 


0.109 


-0.0661 


0.157 


-0.0664 


0.115 


-0.0141 


0.0706 


0.168 


0.0749 


Deviance 


0.163 


0.0689 


0.0258 


0.0236 


0.0307 


0.0514 


0.036 


0.0409 


0.032 


0.14 


0.14 


0.141 


0.14 


0.14 


0.14 


0.141 


0.143 


0.143 


0.141 


0.14 


0.141 


0.141 


0.0693 


0.072 


0.0706 


0.0004 


0.0644 


0.0567 


0.0611 


0.0545 


0.0513 


0.0519 


0.0535 


0.0559 


0.0626 


0.0903 


0.124 


0.0496 


0.0443 


0.0179 


0.0266 


0.0465 


0.0384 


0.0178 


0.119 


0.0232 


0.0216 


0.0203 


0.0171 


-0.885 


-33.5 


2 .18 


-21 . 8 


-2.47 


-4.66 


8.16 


15.7 


4.96 


-1.55 


10 


-0.579 


3.2 


29.4 


0.862 


5.31 


11 


3.8 


6.9 


13.2 


17.4 


21.8 


22.5 


18.1 


12.8 


1.22 


5.58 


5.51 


4.09 


-1.42 


4.08 


-3.73 


0.968 


-0.607 


3.26 


8.25 


4.37 


0.377 


0 


0.0293 


0 


0.0136 


0 


0 


0 


0 


0 . 121 


0.0172 


0.002 


0 


0 


0 


0 


0 


0.0087 


0.0013 


0.0368 


0.0804 


0 . 121 


0 


0.563 


0.0014 


0 


0.389 


0 


0 


0.0002 


0 


0 


0 


0 


0 


0 


0 


0.223 


0 


0 


0 


0.155 


0 


0.0002 


0.333 


0.544 


0.0011 


0 


0 


0.866 


0.0998 


1.06 


0.598 


0.927 


0.787 


1.34 


1.9 


1 . 17 


0.804 


0.716 


0.647 


0.472 


0.435 


0.443 


0.401 


0.453 


0.687 


0.636 


0.746 


0.782 


0.803 


2 


0.959 


1.25 


1.01 


1.06 


1.35 


1.96 


1.23 


1.43 


1.99 


2.54 


3.37 


4.09 


5 . 14 


4.89 


1.06 


1.28 


1 . 1 


1 . 12 


0.936 


1  .  17 


0.936 


1 . 12 


0.986 


1.07 


1.18 


1.08 


Null  Deviance:  131532.1  on  101481  degrees  of  freedom 
Residual  Deviance:  114885.9  on  101433  degrees  of  freedom 
11432  observations  deleted  due  to  missing  values 
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B.  FIRST  COURSE  EFFECTS  MODELS 

In  addition  to  identifying  factors  affecting  course  completions,  we  attempted  to 
estimate  the  effects  of  first  course  results  on  subsequent  attempts.  This  was  done  to 
evaluate  a  failed  first  course’s  potential  as  a  learning  experience  for  the  sailor  and  a 
screening  tool  for  NCPACE.  Though  passing  the  first  course  and  passing  a  subsequent 
course  are  both  correlated  with  a  sailor’s  innate  ability  and  motivation  to  succeed 
academically,  passing  a  subsequent  course  may  also  be  a  function  of  skills  acquired  from 
the  first  course  such  as  effective  study  habits  and  time  management.  Since  NCPACE 
participants  are  working  full  time  aboard  ships,  sometimes  on  non-traditional  work 
schedules,  these  skills  are  especially  of  valid  concern.  The  outcome  of  the  first  course 
can  be  a  valuable  tool  for  both  sailors  and  NCPACE  administrators  to  evaluate  whether 
sailors  have  learned  the  necessary  skills  in  addition  to  having  the  ability  and  motivation 
to  succeed. 

We  used  the  data  set  of  first  and  second  NCPACE  course  attempts  described  in 
the  previous  chapter  to  evaluate  how  a  first  course  affects  a  sailor’s  decision  to  attempt  a 
subsequent  course  and  how  a  first  course  result  affects  a  sailor’s  performance  in  a 
subsequent  course  given  that  the  sailor  decides  to  attempt  a  subsequent  course. 
Approximately  45  percent  of  NCPACE  participants  have  taken  more  than  one  course. 

1.  Dependent  Variables 

We  constructed  two  models  to  address  the  questions  in  the  previous  chapter: 

•  We  predicted  a  sailor’s  decision  to  attempt  a  subsequent  course.  For  this 
model,  the  dependent  variable  is  an  indicator  variable  that  equals  1  if  a 
sailor  attempts  a  second  course  and  0  if  the  sailor  does  not  attempt  a 
second  course. 

•  We  predicted  a  sailor’s  performance  in  a  second  course.  For  this  model, 
the  dependent  variable  is  an  indicator  variable  that  equals  1  if  a  sailor 
completes  the  second  course  successfully  and  0  if  a  sailor  does  not.  This 
model  is  restricted  to  those  who  taken  a  second  course. 
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2.  Independent  Variables 

For  the  two  models  specified  above,  we  considered  the  first  course  performance 
as  an  independent  variable  in  addition  to  the  variables  we  chose  for  the  course  completion 
model.  The  course  performance  variable  equals  1  for  successfully  completed  first 

courses  and  0  for  unsuccessful  first  courses.  The  model  that  evaluates  a  sailor’s  decision 

to  take  a  subsequent  course  used  course  and  sailor  characteristics  associated  with  the  first 
course.  The  model  that  predicts  performance  in  subsequent  courses  used  course  and 
sailor  characteristics  of  the  second  course,  with  the  exception  of  race,  ethnicity,  sex,  and 
marital  status  (which  are  assumed  to  remain  constant  over  time). 

3.  Model  Assumptions,  Model,  and  Model  Diagnostics 

To  predict  a  sailor’s  decision  to  attempt  a  subsequent  course,  we  assume  Attempt 
subsequent  coursei  is  a  Bernoulli  random  variable  with  probability  distribution  as 
follows: 


Attempt  subsequent  course. 

Probability 

1 

P(Attempt  subsequent  course,  =  1)  =  tt, 

0 

P(A ttempt  subsequent  course,  =  0)  =  1  —  n, 

Table  5 .  Probability  Distribution  of  Attempt  subsequent  courset 


n 

We  chose  the  logit  transformation  In  1  —  u  =  fio  +  fin  Course  subjecU  +  fii,  Unit 
typet  +  fin  Fiscal  year,  +  fi4i  School,  +  [is  AFQT  +  fir,,  Educationi  +  fir,  Rank +  fin,  Ratingj 
+  fa  Race i  +  fi w  Ethnicity  +  (in  Sex  +  fin  Successful  completion  of  first  course.  We 
omitted  Course  delivery  method  and  Marital  status  because  the  analysis  of  deviance  table 
corresponding  to  the  model  containing  these  factors  demonstrates  that  these  factors  are 
not  statistically  significant  at  the  .05  level  (Table  6). 
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m 

NULL 

101000 

139000 

0 

Technology 

i 

1 . 67 

101000 

139000 

0.196 

CourseType 

4 

108 

101000 

139000 

0 

UnitType 

3 

261 

101000 

139000 

0 

FY 

13 

2430 

101000 

136000 

0 

School 

3 

94.2 

101000 

136000 

0 

Afqt 

1 

296 

101000 

136000 

0 

Education 

3 

269 

101000 

135000 

0 

Rank 

8 

35.9 

101000 

135000 

1.85E-05 

Rating 

4 

CD 

t — 1 

CXI 

101000 

135000 

0.000242 

Race 

5 

199 

101000 

135000 

0 

Hispanic 

1 

56.5 

101000 

135000 

5.6E-14 

Female 

1 

143 

101000 

135000 

0 

Married 

1 

0.0874 

101000 

135000 

0.767 

Pass 

1 

3790 

101000 

131000 

0 

Table  6.  Attempt  Subsequent  Course  Model  Analysis  of  Deviance  Table 


The  Hosmer-Lemeshow  test  statistic  has  a  p-value  of  0.0413  (Chi-squared  = 
123.5834,  df  =  98),  and  Figure  20  is  a  plot  of  the  observed  versus  expected  values  for  the 
attempt  subsequent  course  model.  Although  the  p-value  of  the  test  statistic  is  smaller 
than  .05,  the  plot  indicates  that  the  model  “somewhat”  fits  the  data.  Figure  21  shows  the 
ROC  curve  for  the  attempt  subsequent  course  model.  The  AUC  is  0.654  and  all  points  lie 
above  the  chance  line,  indicating  that  the  model  predicts  better  than  random  guessing. 


Observed  vs.  Expected 


Figure  20.  Hosmer-Lemeshow  Plot  of  Attempt  Subsequent  Course  Model 
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ROC 


Figure  2 1 .  ROC  Curve  of  Attempt  Subsequent  Course  Model 

To  predict  a  sailor’s  performance  in  a  subsequent  course  given  that  they  decide  to 
attempt  a  subsequent  course,  we  assume  Complete  subsequent  courset  is  a  Bernoulli 
random  variable  with  probability  distribution  as  follows: 


Complete  subsequent  course 

Probability 

1 

P(Complete  subsequent  course,  =  1  Attempt 
subsequent  course ,  =  1)  =  TCj 

0 

P(Complete  subsequent  course,  =  0  Attempt 
subsequent  course ,■  =  1)  =  1  -  7rr- 

Table  7.  Probability  Distribution  of  Complete  subsequent  courset 


ix 

We  used  the  logit  transformation  In  1  -  IX  =  fo  +  [it  Course  delivery  method  +  [hi 
Course  subject ,  +  /C,  Unit  tvpei  +  [Ui  Fiscal  year ,■  +  /A  Schoob  +  [h,  AFQT  +  [hi 
Educationt  +  fist  Rankj  +  [ig,  Ratingj  +  Pm  Race +  [in  Sex  +  P12  Successful  completion  of 
first  course.  We  omitted  Ethnicity  and  Marital  status  because  the  analysis  of  deviance 
table  corresponding  to  the  model  containing  these  factors  demonstrates  that  these  factors 
are  not  statistically  significant  at  the  .05  level  (Table  8). 
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Pr (Chi) 


Df  Deviance  Resid.  Df  Resid.  Dev 


NULL 

101000 

135000 

0 

Technology. subseqCourse 

1 

9550 

101000 

125000 

0 

Course . subseqCourse 

4 

686 

101000 

125000 

0 

UnitType . subseqCourse 

3 

742 

101000 

124000 

0 

FY . subseqCourse 

EO 

1480 

101000 

122000 

0 

School . subseqCourse 

3 

603 

101000 

122000 

0 

Af qt . subseqCourse 

1 

1190 

101000 

121000 

0 

Education . subseqCourse 

3 

782 

101000 

120000 

0 

Rank . subseqCourse 

8 

3010 

101000 

117000 

0 

Rating . subseqCourse 

4 

36.6 

101000 

117000 

0.000000218 

Race 

5 

36.8 

101000 

117000 

0.000000642 

Hispanic 

1 

1.22 

101000 

117000 

0.27 

Female 

1 

33 

101000 

117000 

9.34E-09 

Married 

1 

1 . 77 

101000 

117000 

0.183 

Pass . f irstCourse 

1 

45300 

101000 

71300 

0 

Table  8.  Complete  Subsequent  Course  Model  Analysis  of  Deviance  Table 


The  Hosmer-Lemeshow  test  statistic  has  a  p-value  of  0  (Chi-squared  =  167.7942, 
df  =  98),  and  Figure  22  is  a  plot  of  the  observed  versus  expected  values  for  the  complete 
subsequent  course  model.  Although  the  p-value  of  the  test  statistic  is  small,  the  plot 
indicates  that  the  model  fits  the  data  “reasonably”  well,  though  the  mid-range  observed 
values  are  slightly  lower  than  the  expected  values. 


Observed  vs.  Expected 


0.0  0.2  0.4  0.6  0.8  1.0 

%  Observed  Per  Group 


Figure  22.  Hosmer-Lemeshow  Plot  of  Complete  Subsequent  Course  Model 
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Figure  23  shows  the  ROC  curve  for  the  complete  subsequent  course  model.  We 
have  “outstanding”  classification  results,  as  the  AUC  is  0.906. 

ROC 


Figure  23.  ROC  Curve  of  Complete  Subsequent  Course  Model 

4.  Analysis 

The  results  of  the  attempt  subsequent  course  model  are  presented  in  Table  9. 
We  reject  the  null  model  in  favor  of  this  model,  as  our  test  statistic  7993.7  is  greater  than 
5Ca,47  for  any  reasonable  choice  for  a.  Passing  the  first  course  has  the  effect  of 
multiplying  the  odds  of  completing  the  second  course  by  2.5. 
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(Intercept) 


CourseBus 


CourseMath 


CoursePSci 


-0.334 


-0.233 


0.0589 


-0.0395 


-0.0426 


-0.129 


0.112 


CourseOSci 


UnitTypeOther 


UnitTypeSubmarine 


UnitTypeSurfaceCombat  0.0463 


FY1996 


FY1997 


FY1998 
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SchoolCoastline 


SchoolRichland 


SchoolOther School 
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EdualtHS 


EduHS 


EdusomeColl 


RankE2 


RankE3 


RankE4 


RankE5 


RankE6 


RankE7 


RankE8 


RankE9 


RatingFR 


Ratingln 


RatingMa 


RatingOp 


RaceAIAN 


RaceAsian 


RaceBlack 


RacePI 


RaceOtherRace 


Hispanic 


Female 


-0.532 


-0.434 


-0.438 


-0.273 


-0.291 


-0.719 


-0.743 


-1.32 


-0.801 


-0.68 


-0.808 


-1.02 


-1.7 


0.284 


0.193 


0.137 


0.0039 


-0.0762 


0.024 


0.247 


0.0211 


-0.0972 


-0.204 


-0.221 


-0.185 


-0.219 


-0.206 


-0.363 


-0.143 


-0.0453 


0.0493 


0.0155 


-0.0557 


0.0583 


-0.143 


0.213 


-0.279 


0.138 


0.197 


0.921 


Null  Deviance:  1 
Residual  Deviance: 
11412  observat 


0.145 


0.0232 


0.0226 


0.0306 


0.0531 


0.0331 


0.0387 


0.0293 


0.122 


0.122 


0.122 


0.122 


0.122 


0.122 


0.123 


0.125 


0.125 


0.122 


0.122 


0.122 


0.124 


0.0203 


0.0292 


0.0266 


0.0004 


0.0609 


0.0535 


0.057 


0.0533 


0.0501 


0.0504 


0.0513 


0.0525 


0.0576 


0.0786 


0.107 


0.0448 


0.0391 


0.0164 


0.0244 


0.0434 


0.0347 


0.0163 


0.108 


0.0214 


0.0199 


0.0185 


0.0153 


38953.1  on  10 
131259.4  on 
ions  deleted 


-2.3 


-10 


2.6 


-1.29 


-0.803 


-3.89 


2.89 


1.58 


-4.37 


-3.56 


-3.58 


-2.24 


-2.39 


-5.9 


-6.05 


-10.6 


-6.44 


-5.57 


-6.63 


-8.3 


-13.8 


0.157 


3.76 


16.4 


18.5 


12.4 


14.4 


6.88 


11.5 


10.2 


1.34 


9.04 


0.406 


1 . 64 


2.83 


76.6 


3.89 


170 


48.3 


113 


3630 


1031  degrees  of  freedom 
100984  degrees  of  freedom 
due  to  missing  values 


0.0213 


0 


0.0093 


0.197 


0.422 


0.0001 


0.0039 


0 .114 


0 


0.0004 


0.0004 


0.0251 


0.0168 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0.211 


0.654 


0 


0.692 


0.0525 


0.0001 


0 


0.0004 


0.0002 


0.0087 


0.0007 


0.0014 


0.247 


0.0026 


0.524 


0.2 


0.0927 


0 


0.0485 


0 


0 


0 


0 
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The  results  of  the  complete  subsequent  course  model  are  presented  in  Table  10. 
We  reject  the  null  model  in  favor  of  this  model,  as  our  test  statistic  63477.18  is  greater 
than  x~a,47  for  any  reasonable  choice  for  a.  Passing  the  first  course  has  the  strongest 
effect  on  odds  of  completing  the  second  course.  We  also  found  the  following  similarities 
to  the  first  course  completion  model: 

•  Higher  ranks  have  higher  predicted  success  rates; 

•  Instructor-based  courses  have  higher  predicted  success  rates; 

•  Mathematics  courses  have  lower  predicted  success  rates; 

•  Submarine  units  had  higher  predicted  success  rates; 

•  Higher  AFQT  scores  and  levels  of  education  were  associated  with  higher 
predicted  success  rates;  and 

•  Courses  sponsored  by  Coastline  Community  College  had  higher  predicted 
success  rates. 
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(Intercept) 


Technology. subCrs 


CourseBus . subCrs 


Cour seMath . subCrs 


CoursePSci . subCrs 


CourseOSci . subCrs 


UnitTypeOther . subCrs 


UnitTypeSubmarine . subCrs 


UnitTypeSurf aceCombat . subCrs 


FY1996 . subCrs 


FY1997 . subCrs 


FY1998. subCrs 


FY1999. subCrs 


FY2000. subCrs 


FY2001 . subCrs 


FY2002 . subCrs 


FY2003 . subCrs 


FY2004 . subCrs 


FY2005. subCrs 


FY2006 . subCrs 


FY2007 . subCrs 


FY2008. subCrs 


SchoolCoastline . subCrs 


SchoolRichland. subCrs 


SchoolOtherSchool . subCrs 


Af qt . subCrs . subCrs 


EdualtHS . subCrs 


EduHS . subCrs 


EdusomeColl . subCrs 


RankE2 . subCrs 


RankE3 . subCrs 


RankE4 . subCrs 


RankE5 . subCrs 


RankE6 . subCrs 


RankE7 . subCrs 


RankE8 . subCrs 


RankE9 . subCrs 


RatingFR. subCrs 


Ratingln . subCrs 


RatingMa . subCrs 


RatingOp . subCrs 


RaceAIAN 


RaceAsian 


RaceBlack 


RacePI 


RaceOtherRace 


Female 


Pass . f irstCourse 


-1.73  0.229  -7.56 


-2.15  0.0868  -24.8 


0.124  0.0355  3.49 


-0.587  0.0319  -18.4 


0.0053  0.0432  0.122 


-0.174  0.0693  -2.51 


0.172  0.0495  3.47 


0.438  0.0563  7.78 


-0.0116  0.0443  -0.263 


-0.618  0.195  -3.17 


-0.741  0.194  -3.81 


-0.826  0.195  -4.24 


-0.992  0.194  -5.11 


-1.07  0.194  -5.53 


-1  0.193  -5.17 


-1.02  0.195  -5.23 


-0.901  0.196  -4.59 


-0.468  0.199  -2.35 


-0.665  0.194 


-0.417  0.194 


-0.414  0.194 


-0.226  0.194 


0.72  0.0877  8.21 


-0.0014  0.0918  -0.0157 


0.261  0.0888  2.94 


0.007  0.0006  12.4 


0.113  0.0891  1.27 


0.208  0.0785  2.66 


0.507  0.0839  6.04 


0.206  0.092  2.23 


0.343  0.0867  3.96 


0.579  0.0868  6.67 


0.842  0.0879  9.58 


1.08  0.0895  12 


1.22  0.0958  12.8 


1.35  0.124  10.9 


1.43  0.165  8.68 


0.0874  0.0657  1.33 


0.173  0.0587  2.94 


0.0163  0.0243  0.668 


0.0368  0.0356  1.03 


0.0346  0.0642  0.539 


0.161  0.0522  3.08 


0.005  0.024  0.208 


0.296  0.166  1.78 


0.0694  0.0314  2.21 


0.0243  0.0275  0.885 


3.66  0.0207 


Null  Deviance:  134818.8  on  101034  degrees 
Residual  Deviance:  71341.62  on  100987  degre 
11409  observations  deleted  due  to  missi 


57.2 


615 


12.2 


339 


0.0149 


6.32 


12 . 1 


60.6 


0.0689 


10 


14.5 


17 . 9 


26.1 


30.6 


26.8 


27.4 


21 . 1 


5.51 


11.7 


4.63 


4.57 


1.36 


67.3 


0.0003 


8.63 


154 


1.61 


7.05 


36.5 


4.99 


15.7 


44.5 


91 . 9 


145 


163 


119 


75.4 


1.77 


8.65 


0.447 


1.07 


0.291 


9.5 


0.0433 


3.17 


4.89 


0.784 


31400 


0 


0 


0.0005 


0 


0.903 


0.0119 


0.0005 


0 


0 


0.0015 


0.0001 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0.204 


0 


0 


0.0255 


0.0001 


0 


0 


0 


0 


0 


0 


0.183 


0.0033 


0.504 


0.302 


0.59 


0 


0.835 


0 


0 


0 


0 


1034  degrees  of  freedom 
100987  degrees  of  freedom 
due  to  missing  values 
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c. 


PROMOTION  MODEL 


The  goal  of  the  promotion  model  is  to  determine  if  successfully  completing 
NCPACE  courses  has  an  effect  on  promotion.  This  model  considers  promotions  to  E5 
and  is  restricted  to  NCPACE  participants  with  four-year  contracts  in  their  first  term.  We 
included  16,686  sailors  who  left  before  serving  36  months  of  their  48-month  contract  in 
the  sample  used  for  the  promotion  model. 

1.  Dependent  Variable 

The  dependent  variable  for  this  model  is  an  indicator  variable  that  equals  1  if  the 
sailor  promoted  to  E5  by  the  end  of  their  first  term  and  0  otherwise. 

2.  Independent  Variables 

The  independent  variables  we  chose  are  identical  to  the  variables  chosen  for  the 
Effects  of  First  Course  Results  model  with  the  course-specific  variables  omitted,  an 
indicator  variable  added  which  captures  if  the  participant  succeeded  an  NCPACE  course 
during  their  first  term,  and  fiscal  year  indicator  variables  for  the  date  of  accession  and 
separation. 


3.  Model  Assumptions,  Model,  and  Model  Diagnostics 

To  predict  a  sailor’s  decision  promotion  to  E5,  we  assume  Promoted  is  a 
Bernoulli  random  variable  with  probability  distribution  as  follows: 


Promotedi 

Probability 

1 

P(Promotedj  =  1 )  =  7r, 

0 

PfiPromotedi  =  0)  =  1  —  jtt 

Table  1 1 .  Probability  Distribution  of  Promoted 


7T 

We  used  the  logit  transformation  In  1  -  tr  =  fio  +  fii  Succeeded  at  least  one 
NCPACE  Course  +  [Pi  Unit  type,  +  da  Fiscal  year  of  first  NCPACE  course i  +  [I4  AFQT  + 
[fii  Education +  fia  Rankj  +  fij,  Rating y  +  fist  Racel  +  fig  Marital  status  +  fiio  Fiscal  Year 
of  Accession  +  fin  Fiscal  Year  of  Separation/Reenlistment.  We  omitted  Ethnicity  and 
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Sex  because  the  analysis  of  deviance  table  corresponding  to  the  model  containing  these 
factors  demonstrates  that  these  factors  are  not  statistically  significant  at  the  .1  level 
(Table  12). 


NULL 

41900 

57800 

0 

NCPACEsuccess 

1 

83.5 

41900 

57700 

0 

UnitType 

3 

839 

41900 

56900 

0 

FY 

13 

1190 

41900 

55700 

0 

Afqt 

1 

1890 

41900 

53800 

0 

Education 

3 

518 

41800 

53300 

0 

Rank 

8 

4400 

41800 

48900 

0 

Rating 

4 

610 

41800 

48300 

0 

Race 

5 

2420 

41800 

45900 

0 

Hispanic 

1 

1.31 

41800 

45900 

0.253 

Female 

1 

0.133 

41800 

45900 

0.715 

Married 

1 

7.54 

41800 

45900 

0.00603 

AccFY 

9 

982 

41800 

44900 

0 

SepReenFY 

13 

3770 

41800 

41100 

0 

Table  12.  Promotion  Model  Analysis  of  Deviance  Table 


Although  the  Hosmer-Lemeshow  test  statistic  has  a  p-value  of  0  (Chi-squared  = 
210.6678,  df  =  98),  the  plot  indicates  that  the  model  fits  the  data  “reasonably”  well 
(Figure  24). 


Observed  vs.  Expected 


0.0  0.2  0.4  0.6  0.8  1.0 

%  Observed  Per  Group 


Figure  24.  Hosmer-Lemeshow  Plot  of  Promotion  Model 
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Figure  25  shows  the  ROC  curve  for  the  promotion  model.  We  have  “excellent” 
classification  results,  as  all  points  of  the  ROC  are  above  the  chance  line  and  the  AUC  is 
0.840. 


ROC 


Figure  25.  ROC  Curve  of  Promotion  Model 

4.  Analysis 

The  results  of  the  promotion  model  are  presented  in  Table  13.  We  reject  the  null 
model  in  favor  of  this  model,  as  our  test  statistic  16713.14  is  greater  than  %  a.b i  for  any 
reasonable  choice  for  a.  We  found  that  those  who  successfully  completed  at  least  once 
course  are  predicted  to  be  more  likely  to  promote  to  E5  by  the  end  of  their  first  term. 


Value 

Std.  Error 

t  value 

ChiSq 

Pr (Chi) 

Odds  Ratio 

(Intercept) 

2.83 

0.786 

3.6 

12 . 9 

0.0003 

16.9 

NCPACEsuccess 

0.186 

0.0261 

7.15 

51 . 1 

0 

1.2 

UnitTypeOther 

0.129 

0.0577 

2.24 

5 

0.0253 

1  .  14 

UnitTypeSubmarine 

0.491 

0.0759 

6.46 

41 . 8 

0 

1 . 63 

UnitTypeSurfaceCombat 

-0.0443 

0.0495 

-0.895 

0.801 

0.371 

0.957 

FY1996 

-1.09 

0.772 

-1.41 

1.99 

0.158 

0.336 

FY1997 

-1.62 

0.768 

-2 .11 

4.47 

0.0345 

0.197 

FY1998 

-1 . 94 

0.769 

-2.52 

6.37 

0.0116 

0.144 

FY1999 

-2.39 

0.77 

-3.11 

9.67 

0.0019 

0.0912 

FY2000 

-2.68 

0.772 

-3.48 

12 . 1 

0.0005 

0.0683 

FY2001 

-3.09 

0.773 

-4 

16 

0.0001 

0.0455 

FY2002 

-3.69 

0.776 

-4.76 

22 . 6 

0 

0.0249 

FY2003 

-3.92 

0.778 

-5.04 

25.4 

0 

0.0198 

FY2004 

-4.71 

0.782 

-6.02 

36.3 

0 

0.009 

FY2005 

-4.83 

0.782 

-6.18 

38.2 

0 

0.008 

FY2006 

-5.26 

0.785 

-6.7 

44 . 9 

0 

0.0052 

FY2007 

-5.65 

0.787 

-7 . 17 

51.4 

0 

0.0035 
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Value  Std.  Error  t  value  ChiSq  Pr(Chi)  Odds  Ratio 


FY2008 


Afqt 


EducationaltHS 


EducationHS 


EducationsomeColl 


RankE2 


RankE3 


RankE4 


RankE5 


RankE6 


RankE7 


RankE8 


RankE9 


RatingFR 


Ratingln 


RatingMa 


RatingOp 


RaceAIAN 


RaceAsian 


RaceBlack 


RacePI 


RaceOtherRace 


Married 


AccFYl 996 


AccFYl 997 


AccFYl 998 


AccFYl 999 


ACCFY2000 


ACCFY2001 


ACCFY2002 


ACCFY2003 


ACCFY2004 


SepReenFY1996 


SepReenFY1997 


SepReenFY1998 


SepReenFY1999 


SepReenFY2000 


SepReenFY2001 


SepReenFY2002 


SepReenFY2003 


SepReenFY2004 


SepReenFY2005 


SepReenFY2006 


SepReenFY2007 


SepReenFY2008 


-6.17 


0.012 


-0.237 


0.0301 


0.771 


0.256 


0.59 


1.71 


5.06 


1.57 


1 . 1 


0.751 


2 .19 


-0.51 


0.979 


0.813 


-0.172 


0.011 


-1.19 


-0.151 


-1.59 


0.126 


1.07 


1.65 


2.16 


2.08 


1.78 


1.45 


1.02 


0.627 


0.0324 


-4.55 


-4.51 


-4.21 


-3.42 


-3.32 


-2.96 


-2.87 


-2.3 


-3.55 


-2 . 92 


-1.2 


-0.531 


0.221 


Null  Deviance: 
Residual  Deviance 
5535  observat 


0.797 


0.0007 


0.102 


0.0908 


0.103 


0.077 


0.0742 


0.0795 


0.137 


0.359 


0.501 


0.671 


0.504 


0.132 


0.0808 


0.0294 


0.0453 


0.0769 


0.0635 


0.0314 


0.197 


0.0398 


0.0322 


0.0784 


0.0857 


0.0988 


0.11 


0.125 


0.141 


0.155 


0.165 


0.175 


0.281 


0.251 


0.183 


0.126 


0 . 12 


0.113 


0.11 


0.105 


0.109 


0.105 


0.0972 


0.0955 


0.102 


-7.74 


16.3 


-2.32 


0.331 


7.52 


3.33 


7.95 


21.5 


37 


4.38 


2.2 


1 . 12 


4.34 


-3.85 


12 . 1 


-6.74 


17 . 9 


-2.23 


0.173 


-38 


-0.764 


-39.8 


3.9 


13.6 


19.2 


21 . 9 


18.9 


14.3 


10.3 


6.54 


3.8 


0.185 


-16.2 


-18 


-23.1 


-27 . 1 


-27 . 6 


-26.3 


-26.1 


-21 . 9 


-32.6 


-27.8 


-12.3 


-5.56 


2.16 


57814.77  on  41870  degrees  of  freedom 
:  41101.63  on  41809  degrees  of  freedom 
ions  deleted  due  to  missing  values 


45.5 


322 


4.99 


0.0301 


1450 


0.584 


1590 


15.2 


186 


371 


480 


357 


203 


107 


42.7 


14.4 


0.034 


263 


324 


531 


737 


764 


690 


681 


482 


1060 


775 


152 


30.9 


4.66 


0 


0 


0.0202 


0.74 


0 


0.0009 


0 


0 


0 


0 


0.028 


0.263 


0 


0.0001 


0 


0 


0 


0.0255 


0.862 


0 


0.445 


0 


0.0001 


0 


0 


0 


0 


0 


0 


0 


0.0002 


0.854 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0.0309 


0.0021 


1.01 


0.789 


1.03 


2.16 


1.29 


1.8 


5.55 


158 


4.82 


3 


2 . 12 


8.92 


0.601 


2.66 


0.82 


2.25 


0.842 


1.01 


0.303 


0.86 


0.205 


1 . 13 


2 . 91 


5.21 


8 .71 


8.03 


5.92 


4.28 


2.76 


1 . 87 


1.03 


0.0105 


0.011 


0.0148 


0.0327 


0.036 


0.0517 


0.0568 


0.101 


0.0287 


0.0539 


0.302 


0.588 


1.25 


Table  13.  Promotion  Model  Results 


In  light  of  Buddin  and  Kapur’s  study,  we  examined  the  independent  variable 
Succeeded  at  least  one  NCPACE  Course  in  our  model  to  see  if  it  is  sensitive  to  sailors 
who  attrite  early.  We  reran  the  regression  with  a  sample  that  excluded  those  who  left 
before  serving  36  months  of  their  48-month  contract.  The  resulting  p-value  of  this 
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variable  with  the  modified  sample  was  0.04,  with  a  coefficient  of  .094  (odds  ratio  of 
1.09).  The  promotion  effects  in  the  model  without  early  separators  are  slightly  smaller. 


D.  RETENTION  MODELS 

The  objective  of  the  extension  and  reenlistment  models  are  to  detennine  if  success 
in  NCPACE  affects  a  first  tenn  sailor’s  decision  to  extend  their  contract  beyond  its 
original  term  of  48  months.  Like  the  promotion  model,  the  retention  models  were 
restricted  to  NCPACE  participants  with  four-year  contracts  in  their  first  term  and 
included  sailors  who  left  before  serving  36  months  of  their  48-month  contract. 

1.  Dependent  Variable 

The  dependent  variable  for  the  extension  model  is  an  indicator  variable  which 
equals  1  if  the  sailor  is  still  serving  after  48  months  of  service  and  0  if  the  sailor  did  not. 

The  dependent  variable  for  the  reenlistment  model  is  an  indicator  variable  which 
equals  1  if  the  sailor  has  a  reenlistment  action  on  record  after  accessing  0  if  the  sailor 
does  not. 

2.  Independent  Variables 

The  independent  variables  we  chose  are  identical  to  the  variables  chosen  for  the 
promotion  model. 

3.  Model  Assumptions,  Model,  and  Model  Diagnostics 

To  predict  a  sailor’s  decision  to  reenlist,  we  assume  Reenlisti  is  a  Bernoulli 
random  variable  with  probability  distribution  as  follows: 


Reenlisti 

Probability 

1 

P{  Reenlisti  =  \  )  =  n, 

0 

P{ReenIisti  =  0)  =  1  -  jtj 

Table  14.  Probability  Distribution  of  Reenlisti 
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7T 

For  the  extension  model,  we  used  the  logit  transformation  In  1  -  v.  =  fio  +  fii 
Succeeded  at  least  one  NCPACE  Course  +  fci  Unit  type,  +  fiy,  Fiscal  year  of  first 
NCPACE  courset  +  ff  AFQT  +  fly,  Education,  +  fi6i  Ranki  +  fir,  Rating,  +  fist  Race,  +  fig 
Ethnicity  +  ffo  Sex  +  fin  Marital  status  +  fin  Fiscal  Year  of  Accession  +  fin  Fiscal  Year 
of  Separation.  The  analysis  of  deviance  table  corresponding  to  the  course  completion 
model  demonstrates  that  the  selected  factors  are  statistically  significant  at  the  .05  level 
(Table  15). 


Df 

Deviance 

Resid.  Df 

Resid.  Dev 

Pr (Chi) 

NULL 

41900 

55100 

0 

NCPACEsuccess 

1 

4.88 

41900 

55100 

0.0272 

UnitType 

3 

211 

41900 

54900 

0 

FY 

13 

14200 

41900 

40700 

0 

Afqt 

1 

418 

41900 

40300 

0 

Education 

3 

319 

41800 

40000 

0 

Rank 

8 

3060 

41800 

36900 

0 

Rating 

4 

79 

41800 

36800 

3.33E-16 

Race 

5 

2010 

41800 

34800 

0 

Hispanic 

1 

8 . 17 

41800 

34800 

0.00426 

Female 

1 

49.3 

41800 

34800 

2 . 15E-12 

Married 

1 

42.3 

41800 

34700 

7 . 98E-11 

AccFY 

9 

1950 

41800 

32800 

0 

SepReenFY 

13 

3350 

41800 

29400 

0 

Table  15.  Extension  Model  Analysis  of  Deviance  Table 


The  Hosmer-Lemeshow  test  statistic  has  a  p-value  of  0  (Chi-squared  =  950.4458, 
df  =  98),  and  Figure  26  is  a  plot  of  the  observed  versus  expected  values  for  the  extension 
model.  The  plot  shows  that  the  model  “somewhat”  fits  the  data,  though  mid-range 
observed  values  are  slightly  higher  than  the  expected  values  and  the  upper-range 
observed  values  are  slightly  lower  than  the  expected  values. 
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Observed  vs.  Expected 


0.0  0.2  0.4  0.6  0.8  1.0 

%  Observed  Per  Group 


Figure  26.  Hosmer-Lemeshow  Plot  of  Extension  Model 

Figure  27  shows  the  ROC  curve  for  the  extension  model.  We  have  “outstanding” 
classification  results,  as  all  points  of  the  ROC  are  above  the  chance  line  and  the  AUC  is 
0.917. 


ROC 


Figure  27.  ROC  Curve  of  Extension  Model 

7T 

For  the  reenlistment  model,  we  used  the  logit  transformation  In  1  -  7T  =  fo  +  f>  / 
Succeeded  at  least  one  NCPACE  Course  +  p2i  Unit  type,  +  /fi,  Fiscal  year  of  first 
NCPACE  coursei  +  fa  Education  +  ffn  Ranki  +  /fi,  Ratingi  +  /?&■  Race,  +  fw  Sex  +  fin 
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Marita l  status  +  f  12  Fiscal  Year  of  Accession  +  fi  13  Fiscal  Year  of  Separation.  We 
omitted  Afqt  and  Ethnicity  because  the  analysis  of  deviance  table  corresponding  to  the 
model  containing  these  factors  demonstrates  that  these  factors  are  not  statistically 
significant  at  the  .05  level  (Table  16). 


Df 

Deviance 

Resid.  Df 

Resid.  Dev 

Pr (Chi) 

NULL 

41900 

50200 

0 

NCPACEsuccess 

1 

16.9 

41900 

50200 

3.98E-05 

UnitType 

3 

303 

41900 

49900 

0 

FY 

EH 

3270 

41900 

46600 

0 

Afqt 

1 

1.4 

41900 

46600 

0.237 

Education 

3 

450 

41800 

46200 

0 

Rank 

8 

1410 

41800 

44800 

0 

Rating 

4 

14.5 

41800 

44800 

0.00592 

Race 

5 

3060 

41800 

41700 

0 

Hispanic 

1 

0.122 

41800 

41700 

0.727 

Female 

1 

1 — 1 

LO 

t — 1 

41800 

41700 

0.000104 

Married 

1 

157 

41800 

41500 

0 

AccFY 

9 

1880 

41800 

39600 

0 

SepReenFY 

13 

7730 

41800 

31900 

0 

Table  16.  Reenlistment  Model  Analysis  of  Deviance  Table 


The  Hosmer-Lemeshow  test  statistic  has  a  p-value  of  0  (Chi-squared  =  1313.531, 
df  =  98),  and  Figure  28  is  a  plot  of  the  observed  versus  expected  values  for  the 
reenlistment  model.  The  plot  reveals  some  systematic  under-  and  over-predicting. 


Observed  vs.  Expected 
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Figure  28.  Hosmer-Lemeshow  Plot  of  Reenlistment  Model 
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Figure  29  shows  the  ROC  curve  for  the  reenlistment  model.  We  have  “excellent” 
classification  results,  as  all  points  of  the  ROC  are  above  the  chance  line  and  the  AUC  is 
0.876. 


ROC 


Figure  29.  ROC  Curve  of  Reenlistment  Model 

4.  Analysis 

The  results  of  the  extension  model  are  presented  in  Table  17.  We  reject  the  null 
model  in  favor  of  this  model,  as  our  test  statistic  25699.82  is  greater  than  %  a,6i  for  any 
reasonable  choice  for  a.  We  found  that  completing  at  least  once  course  does  not  have  a 
statistically  significant  effect  on  extending,  as  the  p-value  associated  with  this  factor 
exceeds  any  reasonable  cutoff. 


Value 

Std.  Error 

t  value 

ChiSq 

Pr (Chi) 

Odds  Ratio 

(Intercept) 

5.4 

0.974 

5.55 

30.8 

0 

222 

NCPACEsuccess 

-0.0112 

0.0312 

-0.359 

0.129 

0.72 

0.989 

UnitTypeOther 

-0.204 

0.0678 

-3.01 

9.06 

0.0026 

0.815 

UnitTypeSubmarine 

0.0819 

0.0854 

0.959 

0.92 

0.337 

1.09 

UnitTypeSurfaceCombat 

-0.104 

0.0568 

-1.83 

3.35 

0.0671 

0.901 

FY1996 

0.734 

0.955 

0.769 

0.591 

0.442 

2.08 

FY1997 

0.698 

0.953 

0.733 

0.537 

0.464 

2.01 

FY1998 

0.864 

0.954 

0.906 

0.821 

0.365 

2.37 

FY1999 

0.932 

0.955 

0.976 

0.953 

0.329 

2.54 

FY2000 

1.07 

0.956 

1 . 12 

1.26 

0.263 

2 . 92 

FY2001 

0.807 

0.958 

0.843 

0.711 

0.399 

2.24 

FY2002 

0.424 

0.96 

0.442 

0.195 

0.659 

1.53 

FY2003 

0.218 

0.962 

0.227 

0.0513 

0.821 

1.24 

FY2004 

-0.285 

0.966 

-0.295 

0.0871 

0.768 

0.752 

FY2005 

-0.617 

0.968 

-0.638 

0.407 

0.524 

0.54 
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FY2006 _ 

FY2007 _ 

FY2008 _ 

Afqt _ 

Educational tHS 

EducationHS 

EducationsomeColl 

RankE2 

RankE3 

RankE4 

RankE5 

RankE6 

RankE7 

RankE8 

RankE9 

RatingFR 

Ratingln 

RatingMa 

RatingOp _ 

RaceAIAN 

RaceAsian 

RaceBlack 

RacePI 

RaceOtherRace _ 

Hispanic 

Female 

Married 

AccFY1996 _ 

AccFY1997 _ 

AccFY1998 _ 

AccFY1999 _ 

AccFY2  0  0  0 _ 

AccFY2  0  0 1 _ 

AccFY2  0  02 _ 

AccFY2  0  03 _ 

AccFY2  0  0  4 _ 

SepReenFYl 996 
SepReenFYl 997 
SepReenFYl 998 
SepReenFYl 999 
SepReenFY2000 
SepReenFY2001 
SepReenFY2002 
SepReenFY2003 
SepReenFY2004 
SepReenFY2005 
SepReenFY200  6 
SepReenFY2007 
SepReenFY2008 


Value  Std.  Error  t  value  ChiSq  Pr(Chi) 


-1.03  0.975 _ -1.06  1.13  0.289 

-1.34  0.994  -1.35  1.81  0.178 

-1.67  1.15  -1.45  2.11  0.146 

-0.0021  0.0009 _ -2.35  5.51  0.0189 

0.0004  0.122 _ 0,0029  0 _ 0.998 

-0.027  0.109  -0.247  0.0611  0.805 

0.604  0.121  4.98  24.8  0 

0.0043  0.0909 _ 0.0468  0.0022  0.963 

0.223  0.0875 _ 2,55 _ 6.49  0.0108 

0.946  0.0931 _ 10.2 _ 103 _ 0 _ 

1.51 _ 0.11 _ 13.8 _ 190 _ 0 _ 

1.15 _ 0.388 _ 2.98 _ 8.88  0.0029 

0.326  0.544 _ 0.598  0.358  0.55 

0.817  0.657 _ 1.24 _ 1.54  0.214 

2.31 _ 0 . 805 _ 2 .86 _ 8^2 _ 0.0042 

0.191  0.195 _ 0.983  0.967  0.326 

-0.527  0.0893 _ -5.9 _ 34.9  0 _ 

-0.0725  0.0362  -2  4.02  0.045 

-0.274  0.0517  -5.29  28  0 

-0.0199  0.109 _ -0.183  0.0333  0.855 

0.179  0.084 _ 2 . 14 _ 4.56  0.0328 

-1.54  0.041 _ -37.7  1420  0 _ 

0,46 _ 0.415 _ 1 .11 _ 1.23  0.268 

-1.82  0.0479 _ -38 _ 1450  0 _ 

0.0874  0.0424 _ 2 ,06 _ 4.25  0.0393 

0.297  0.0396 _ 7,49 _ 56.1  0 _ 

0.295  0.0412 _ 7 .16 _ 51.3  0 _ 

-0.573  0.0837 _ -6.84  46.8  0 _ 

-1.37  0.097 _ -14.2  201 _ 0 _ 

-1.88  0.113 _ -16.6  275 _ 0 _ 

-3.56  0.13 _ -27.4  750 _ 0 _ 

-5.79  0.148 _ -39.1  1530  0 _ 

-6.3 _ 0.165 _ -38.1  1450  0 _ 

-7.29  0.188 _ -38.8  1510  0 _ 

-8.09  0.213 _ -37.9  1440  0 _ 

-9.67  0.275 _ -35.2  1240  0 _ 

-5.78  0.25 _ -23.1  534 _ 0 _ 

-5.52  0.219 _ -25.2  635 _ 0 _ 

-5.7 _ 0.186 _ -30.7  940 _ 0 _ 

-5.3 _ 0.143 _ -37.2  1380  0 _ 

-4.44  0.133  -33.4  1110  0 

-3.39  0.123  -27.6  762  0 

-2.74  0.117  -23.4  546  0 

-1.48  0.105  -14  197  0 

-2.36  0.113 _ -20,9  439 _ 0 _ 

-2.28  0.116 _ -19.6  386 _ 0 _ 

-0.878  0.097 _ -9.06  82 _ 0 _ 

-0.492  0.0981 _ -5.01  25.1  0 _ 

0.148  0.101  1.47  2.15  0.143 


Null  Deviance:  55134.94  on  41870  degrees  of  freedom 
Residual  Deviance:  29435.12  on  41807  degrees  of  freedom 
5535  observations  deleted  due  to  missing  values 
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0. 

0285 
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0. 

0018 

0. 

0007 

0. 

0003 

0. 

0001 

0. 

0031 

0. 

004 

0. 

0033 

0. 

005 

0. 

0117 

0. 

0336 

0. 

0648 

0. 

229 

0. 

0942 

0. 
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0. 

416 

0. 

612 

1 . 

16 

Table  17.  Extension  Model  Results 
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The  results  of  the  reenlistment  model  are  presented  in  Table  18.  We  reject  the 
null  model  in  favor  of  this  model,  as  our  test  statistic  18459.6  is  greater  than  Xa,6i  for  any 
reasonable  choice  for  a.  We  found  that  those  who  successfully  completed  at  least  once 
course  are  predicted  to  be  slightly  less  likely  to  reenlist  in  the  Navy. 


t  value 

ChiSq 

Pr (Chi) 

Odds  Ratio 

(Intercept) 

-16.6 

61.4 

-0.271 

0.0733 

0.787 

0 

NCPACEsuccess 

-0.0825 

0.0272 

-3.03 

9.2 

0.00241 

0.921 

UnitTypeOther 

-0.18 

0.0598 

-3.01 

9.03 

0.00265 

0.836 

UnitTypeSubmarine 

0.367 

0.0728 

5.05 

25.5 

0 

1.44 

UnitTypeSurfaceCombat 

-0.145 

0.0512 

-2.83 

8.03 

0.00459 

0.865 

FY1996 

0.698 

1 . 1 

0.632 

0.4 

0.527 

2.01 

FY1997 

0.556 

1 . 1 

0.505 

0.255 

0.614 

1 . 74 

FY1998 

0.619 

1 . 1 

0.562 

0.315 

0.574 

1.86 

FY1999 

0.698 

1 . 1 

0.633 

0.401 

0.527 

2.01 

FY2000 

0.794 

1 . 1 

0.72 

0.518 

0.472 

2.21 

FY2001 

0.835 

1 . 1 

0.756 

0.571 

0.45 

2.3 

FY2002 

0.626 

1 .11 

0.566 

0.32 

0.571 

1.87 

FY2003 

0.668 

1 .11 

0.603 

0.363 

0.547 

1 . 95 

FY2004 

0.364 

1 .11 

0.328 

0.108 

0.743 

1.44 

FY2005 

0.226 

1 .11 

0.203 

0.0412 

0.839 

1.25 

FY2006 

0.423 

1 .11 

0.38 

0.144 

0.704 

1.53 

FY2007 

0.459 

1 . 12 

0.411 

0.169 

0.681 

1.58 

FY2008 

1 . 69 

1.23 

1.38 

1.91 

0.167 

5.44 

EducationaltHS 

-0.132 

0.101 

-1.3 

1 . 68 

0.195 

0.877 

EducationHS 

-0.0437 

0.0887 

-0.492 

0.242 

0.623 

0.957 

EducationsomeColl 

0.864 

0.099 

8.73 

76.2 

0 

2.37 

RankE2 

0.0636 

0.0848 

0.75 

0.563 

0.453 

1.07 

RankE3 

0.164 

0.0813 

2.02 

4.09 

0.0432 

1.18 

RankE4 

0.541 

0.0853 

6.34 

40.2 

0 

1.72 

RankE5 

0.936 

0.0967 

9.68 

93.7 

0 

2.55 

RankE6 

0.526 

0.348 

1.51 

2.28 

0.131 

1.69 

RankE7 

0.478 

0.493 

0.969 

0.938 

0.333 

1.61 

RankE8 

0.526 

0.531 

0.99 

0.981 

0.322 

1.69 

RankE9 

0.948 

0.427 

2.22 

4 . 92 

0.0265 

2.58 

RatingFR 

-0.0475 

0.147 

-0.322 

0.104 

0.747 

0.954 

Ratingln 

-0.118 

0.0757 

-1.56 

2.45 

0.118 

0.888 

RatingMa 

-0.283 

0.0313 

-9.05 

81.9 

0 

0.753 

RatingOp 

-0.2 

0.0455 

-4.39 

19.3 

0.00001 

0.819 

RaceAIAN 

-0.0528 

0.0866 

-0.61 

0.373 

0.542 

0.949 

RaceAsian 

0.194 

0.0669 

2.9 

8.4 

0.00376 

1.21 

RaceBlack 

-1.43 

0.0316 

-45.4 

2060 

0 

0.238 

RacePI 

0.125 

0.281 

0.446 

0.199 

0.656 

1.13 

RaceOtherRace 

-2 .18 

0.0424 

-51.4 

2640 

0 

0.113 

Female 

0.187 

0.0341 

5.5 

30.2 

0 

1.21 

Married 

0.425 

0.0335 

12.7 

161 

0 

1.53 

AccFYl 996 

-0.0908 

0.0711 

-1.28 

1.63 

0.201 

0.913 

AccFYl 997 

-0.107 

0.0773 

-1.38 

1.91 

0.167 

0.899 

AccFYl 998 

0.117 

0.0894 

1.31 

1.72 

0.19 

1 . 12 

AccFYl 999 

-0.186 

0.101 

-1.83 

3.36 

0.067 

0.83 

ACCFY2000 

-0.608 

0.12 

00 

o 

LO 

i 

25.8 

0 

0.544 

ACCFY2001 

-0.501 

0.138 

-3.63 

13.1 

0.00029 

0.606 

ACCFY2002 

-0.367 

0.152 

-2.42 

LO 

00 

LO 

0.0156 

0.693 

ACCFY2003 

-0.839 

0.166 

LO 

O 

LO 

1 

25.5 

0 

0.432 

ACCFY2004 

-1.37 

0.202 

-6.81 

46.4 

0 

0.253 

SepReenFYl 996 

0.431 

188 

0.00229 

0.00001 

0.998 

1.54 

SepReenFYl 997 

14 . 6 

61.4 

0.238 

0.0567 

0.812 

2210000 
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SepReenFYl 998 

15.5 

61.4 

0.253 

0.0641 

0.8 

5610000 

SepReenFYl 999 

15.9 

61.4 

0.259 

0.0672 

0.795 

8160000 

SepReenFY2000 

16.5 

61.4 

0.268 

0.0721 

0.788 

14400000 

SepReenFY2001 

16.7 

61.4 

0.271 

0.0737 

0.786 

17200000 

SepReenFY2002 

16.5 

61.4 

0.269 

0.0726 

0.788 

15200000 

SepReenFY2003 

16.9 

61.4 

0.275 

0.0755 

0.784 

21100000 

SepReenFY2004 

0.0462 

74 . 1 

0.00062 

0 

1 

LO 

O 

i — 1 

SepReenFY2005 

13.7 

61.4 

0.223 

0.0496 

0.824 

867000 

SepReenFY2006 

16.6 

61.4 

0.27 

0.0728 

0.787 

15600000 

SepReenFY2007 

00 

LO 
\ — 1 

61.4 

0.257 

0.0661 

0.797 

7110000 

SepReenFY2008 

|  0.528  | 

66.6 

0.00793 

0.00006 

0.994 

1.7 

1  Null  Deviance: 

55681.5  on  47143  degrees  of  freedom 

Residual 

Deviance 

:  37221.9  on 

47082  degrees  of  freedom 

262 

observations  deleted  due  to  missing  values 

Table  18.  Reenlistment  Model  Results 


In  a  manner  similar  to  the  promotion  model,  we  examined  the  independent 
variable  Succeeded  at  least  one  NCPACE  Course  in  the  reenlistment  model  to  see  if  it  is 
sensitive  to  sailors  who  attrite  before  they  are  able  to  make  a  decision  to  reenlist.  We 
reran  the  regression  with  a  sample  that  excluded  those  who  left  before  serving  36  months 
of  their  48-month  contract.  The  resulting  p-value  of  this  variable  with  the  modified 
sample  was  0,  with  a  coefficient  of  -.141  (odds  ratio  of  .868).  The  negative  reenlistment 
effects  in  the  model  without  early  separators  are  slightly  larger. 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1.  Factors  Associated  with  Successful  NCPACE  Course  Completions 

We  evaluated  the  likelihood  of  successful  first  course  completions  by  course  and 
sailor  attributes.  The  descriptive  statistics  and  multivariate  analysis  demonstrated  the 
following: 

•  Instructor-based  courses  have  higher  predicted  success  rates.  We  found 
that  participants  who  take  Instructor  PACE  courses  were  about  ten  times 
more  likely  to  succeed  than  those  taking  Technology  PACE  courses. 

•  Success  rates  also  vary  across  course  subjects.  Mathematics  courses  had 
the  lowest  predicted  success  rates. 

•  AFQT  scores  and  levels  of  education  were  associated  with  higher 
predicted  success  rates.  Participants  who  have  taken  some  college  courses 
were  twice  as  likely  to  succeed  compared  to  those  who  did  not  finish  high 
school. 

•  Higher  ranked  sailors  succeeded  at  higher  rates  than  lower  ranked  sailors 
E4s  were  twice  as  likely  to  successfully  complete  their  first  course 
compared  to  Els.  E6s  were  about  three  times  more  likely  to  complete 
compared  to  Els. 

•  There  is  some  variance  in  success  by  unit  type.  Submarine  units  have 
higher  predicted  success  rates. 

•  Courses  sponsored  by  Coastline  Community  College  have  higher 
predicted  success  rates. 

2.  Impact  of  First  Course  Results 

We  found  that  the  successful  completion  of  the  first  course  is  positively 
associated  with  enrollments  in  subsequent  courses  and  a  factor  in  successfully  completing 
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subsequent  ones.  Successfully  completing  the  first  course  had  the  biggest  effect  on  both 
the  odds  of  attempting  a  subsequent  course  and  successfully  completing  the  subsequent 
course.  We  also  found  that  the  independent  variables  which  explained  first  course 
completions  also  explain  subsequent  course  completions. 

3.  Impact  of  Successful  NCPACE  Course  Completions  on  Promotion 
and  Retention 

We  evaluated  promotion,  extension  and  reenlistment  outcomes  for  first-term 
NCPACE  participants  with  48-month  contracts.  We  found  that  those  who  successfully 
completed  at  least  once  course  are  predicted  to  be  more  likely  to  promote  to  E5  by  the 
end  of  their  first  term,  even  after  controlling  for  factors  such  as  rating. 

We  found  that  completing  at  least  once  course  does  not  have  a  statistically 
significant  effect  on  extensions.  However,  those  who  successfully  completed  at  least 
once  course  are  predicted  to  be  slightly  less  likely  to  reenlist  in  the  Navy. 

B.  RECOMMENDATIONS 

Throughout  this  thesis,  we  have  identified  factors  that  are  associated  with 
successfully  completing  the  first  NCPACE  course  and  attempting  and  completing 
subsequent  courses.  We  recommend  using  this  infonnation  as  a  basis  for  creating 
targeted  screening  guidelines  and  tailoring  academic  counseling  efforts. 

NCPACE  policies  and  guidelines  continually  emphasize  the  role  of  academic 
counseling  and  command  leadership  in  ensuring  that  participants  succeed  in  the  program. 
While  the  data  we  were  able  to  obtain  explained  some  of  the  variance  in  course 
performance,  we  were  not  able  to  measure  the  effectiveness  of  academic  counseling  and 
command  support.  We  recommend  that  NCPACE  collect,  store,  and  make  available  the 
following  data  for  future  evaluations  of  the  effectiveness  of  the  program: 

•  pre-NCPACE  assessment  test  scores 

•  the  participant’s  education  plan  status  at  the  time  of  the  course  including: 

o  whether  the  participant  has  agreed  to  an  education  plan 
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o  whether  the  course  taken  fulfills  a  requirement  for  a  degree 

•  degree  completion  information 

•  exit  survey  data  after  each  course  to  capture  the  following  qualitative 
factors: 

o  level  of  support  offered  by  the  command  leadership 
o  perceived  quality  of  instructor  and  course  materials  support 
o  reason  for  completing  or  withdrawing. 
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